SoC: AtomPub Week 10 Status

Aug 2, 2008 at 11:57pm in Code

I have great news this week. I have finally figured out the whole Movable Type authentication debacle.

The WSSE Authentication Problem

I want to preface my findings with this advisory. I do not have a full understanding of Perl, and my statements may be wrong. If anyone can correct me, please do. I would rather be told I’m completely wrong than continue down the wrong path.

Late last night while inspecting MT’s Atom API code, I found the troublesome line of code (/lib/MT/AtomServer.pm, line 168-170):

if ($nonce_record && $nonce_record->id eq $auth->{Nonce}) {
        return $app->auth_failure(403, "Nonce already used");
}

When commenting out this line of code, the authentication passed, and I was finally seeing a list of posts stored in Movable Type’s database. Once I realized this was the problematic line, I started investigating what this line is doing. From my limited understanding of Perl, this line is checking to see if the nonce has been used in the past. If it has, the check fails to authenticate.

Once I realized this, I looked at my nonces being sent. I verified each nonce generated was different from the last one, eliminating the possibility of the problem being on my end. This tells me somehow Movable Type is incorrectly adding the current nonce to the list of previous nonces before performing that check.

Looking at the slightly bigger picture:

my $app = shift;
my $auth = $app->get_auth_info
    or return $app->auth_failure(401, "Unauthorized");
for my $f (qw( Username PasswordDigest Nonce Created )) {
    return $app->auth_failure(400, "X-WSSE requires $f")
        unless $auth->{$f};
}
require MT::Session;
my $nonce_record = MT::Session->load($auth->{Nonce});

if ($nonce_record && $nonce_record->id eq $auth->{Nonce}) {
    return $app->auth_failure(403, "Nonce already used");
}
$nonce_record = new MT::Session();
$nonce_record->set_values({
    id => $auth->{Nonce},
    start => time,
    kind => 'AN'
});
$nonce_record->save();

From what I’m understanding, the code appears to add the nonce to the record after checking, however, the nonce is passed through MT::Session->load before the check is performed, indicating that may the source of the issue. Sure enough, MT::Session->load is the problem. After replacing the passed nonce with a blank string, the nonce is no longer in the list of previous nonces and the check passes.

I have notified Movable Type’s developers of this problem, and I hope to hear back soon. The good news is once this error is resolved in Movable Type, the importer should work wonders. The bad news is it may be a bit until the next version of MT is released or the bug is fixed. I’ll try to keep everyone updated on the status as I hear back from the developers.

Future Branches of the Atom Importer

Beyond working with Movable Type, after talking with Lloyd this week, we have decided to look into the possibility of porting the importer to other platforms; namely Vox and WordPress itself. We talked about Vox as there is currently no importer at all, as Vox lacks export methods and APIs such as XML-RPC. The idea for WordPress came up because we thought it might be a great way to move a blog from location to location without having to deal with export files.

Over the next week I will be looking at creating those branches of the importer. Thankfully, since the Atom API is a standard, I should be able to quickly and easily make the ports. We’ll see how that goes next week.

2 Comments

1 Mark Paschal on Aug 5, 2008 at 7:58 pm (Quote):

Hi, Ronald,

Thanks for reporting your problem with WSSE authentication in Movable Type’s AtomPub implementation. As you saw, I’m investigating it as case #80915.

I ran some initial tests against our WSSE implementation, but it seems to check out fine. All the existing AtomPub implementations I know of (as well as one I whipped up to try) worked as designed:

a new, previously unused nonce is not found in the db, and so works
a previously used nonce is found in the db, and we reject the request

The MT::Session::load() function is our ORM’s regular record loader, which works fine for session records in MT, as well as loading all the database objects in all our other products (and in its own unit tests). To be clear, you’re right that we’re using MT::Session::load() here to prevent the use of duplicate nonces: we’re looking to see if there are session records with an id field equal to the nonce the client submitted. If there aren’t any, the request continues and such a session is saved; if we find such a record, we reject the nonce as previously used. The session records are later retired as we retire session records, so we don’t have to keep the records around.

While your suggested fix (look for the empty string instead of the specified nonce) will work in your case, it would also allow anyone else to reuse nonces. That is, disabling the check works for (1), but at the cost of breaking (2).

Under the temporary assumption that MT’s WSSE nonce checking was OK, I looked at the client code in the TypePad importer you mentioned.

After adapting it minimally to work with Movable Type instead, I found that the very first AtomPub request (the $ar->exec() in AtomAPI’s verify_auth() method, called from the constructor) goes through fine without the nonce error. However, by the time Typepad_Import::parse_atom() tries to parse the feed response into entries, the response has become the “Nonce already used” error document. See the first patch for the changes I tried.

The patch also includes some logging in AtomRequest::exec(), reporting the request URL, Authorization header, and the call hierarchy each time it’s invoked. According to my logging, for each attempted import, AtomRequest::exec() is making three AtomPub requests with the same nonce by the time we get to Typepad_Import::parse_atom(). These lines are in the log file. The backtraces say the requests are made by:

Typepad_Import::fetch_atom

line 100 calls AtomAPI::init

line 90 calls AtomAPI::verify_auth

line 158 calls AtomRequest::exec directly
line 162 calls AtomAPI::get_feeds

line 193 calls AtomRequest::exec

line 108 calls AtomAPI::get_feeds

line 193 calls AtomRequest::exec

This makes sense to me, as the only place the nonce is generated is in WSSE::rebuild(), and that’s only called in the WSSE constructor, so this version of the AtomAPI module only generates a nonce once per AtomAPI instance. Judging from that, I was able to fix the error with code as shown in the second patch; Typepad_Import::fetch_atom() then reported getting the correct feed content.

Are you using the same AtomAPI module in the same way in your MT importer? If not, I’ll need more current code from you to further debug the app against our WSSE implementation.

I hope this helps. Thanks again for reporting the case!
2 Ronald Heft on Aug 5, 2008 at 10:23 pm (Quote):

Hi Mark,

Thank you for taking the time to look over the Atom importer. What you outlined makes perfect sense. The same nonce was being sent over multiple request, generating that message. I’m not sure why I didn’t think to check for that. I guess I’ve been attacking this from one angle for too long.

I am indeed using the same AtomAPI class for the Movable Type importer, so that is subject to same bug. I will patch up the AtomAPI class per your suggestions tomorrow morning and let you know if I run into any problems. I don’t suspect I will, and everything should run smoothly from here.

Thanks again for your time, and I’m sorry I sent you on a wild goose chase over a simple repetitive nonce.

6 Trackbacks/Pingbacks

Week 10 status report is now available. … « WordPress Summer of Code (WPSoC) Prologue
on Aug 3, 2008 at 12:01 am

[...] am on August 3, 2008 | # | Tags: atompub importer, movable type, weekly Week 10 status report is now available. I have a full explanation of the WSSE problem and my findings in MT. [...]
The WSSE authentication sage comes to cl … « WordPress Summer of Code (WPSoC) Prologue
on Aug 5, 2008 at 10:25 pm

[...] am on August 6, 2008 | # | Tags: atompub importer, movable type The WSSE authentication sage comes to close. Final verdict? I overlooked an error in the nonce generation of the AtomAPI I was using. [...]
SoC: AtomPub Week 11 Status · cavemonkey50.com
on Aug 10, 2008 at 5:27 pm

[...] received a great deal of help from two Movable Type developers, Mark Paschal and Byrne Reese. After finding the source of my issue, Mark took a closer look at the specific problem. His diagnosis: I had a bug in my end of the WSSE [...]
SoC: AtomPub Week 11 Status | The Booty Blogger
on Sep 12, 2008 at 4:22 pm

[...] received a great deal of help from two Movable Type developers, Mark Paschal and Byrne Reese. After finding the source of my issue, Mark took a closer look at the specific problem. His diagnosis: I had a bug in my end of the WSSE [...]
The Booty Blogger » SoC: AtomPub Week 11 Status
on Sep 14, 2008 at 10:58 am

[...] received a great deal of help from two Movable Type developers, Mark Paschal and Byrne Reese. After finding the source of my issue, Mark took a closer look at the specific problem. His diagnosis: I had a bug in my end of the WSSE [...]
Recent Links Tagged With "wsse" - JabberTags
on Nov 8, 2008 at 12:19 am

[...] public links >> wsse SoC: AtomPub Week 10 Status Saved by tothy0411 on Fri 07-11-2008 [ActionScript3.0] MD5 SHA1??????????????????????????????… [...]