How a sentence is understood

A command goes through six hands between the keyboard and the thing being done: the line reader (READ_LINE), the tokeniser (TOKENISE), the parser's state machine (PARSE_COMMAND), the pattern matcher (MATCH_PATTERN), the object matcher (MATCH_AND_TRY) and the one routine that does anything for anyone (DO_ACTION). The overview says what each is for; this page follows one real line through all of them, with what each left in its buffers.

The line followed

Typed at the first prompt of a game, in Bag End, where Thorin and Gandalf are and the player holds the curious map (Gandalf's first step gives it):

SAY TO THORIN "QUICKLY OPEN THE CHEST" THEN READ THE CURIOUS MAP AND DROP IT

It has an order in quotes with an adverb in it, THEN, two verbs that are synonyms (SAY, READ), an adjective, AND in front of a verb, and IT. The game was run in SkoolKit's simulator when this page was built, the line put into the input buffer as scripts/hobbit_drive.py does, and every table below was read from the machine by a probe on the address named, just before the instruction there ran. What the game printed, from PRINT_CHAR past its gate (the player's commands, then the rest of the world's turns, Thorin's among them):

you talk to thorin.

you examine the curious map.
there seem to be some symbols on it but
you cannot read them.

you drop the curious map.

gandalf opens the round green door.
thorin opens the wooden chest.
gandalf goes east.
thorin says " hurry up ".

1. The line

READ_LINE takes keys into INPUT_LINE as plain ASCII, echoing them, and ends the line with a carriage return ($0D). Nothing is done to the words yet. The cursor is only registers -- HL walking the line, B counting the room left, 128 to start -- which is how a command can be put in from outside (see the notes' driving.md).

2. Words into tokens

The main loop calls TOKENISE once per word, from $6D31, and keeps each two-byte token in TOKENS. A full stop, a comma or a quote is a token by itself (PUNCTUATION_TOKEN). Anything else is copied as letter codes -- the low five bits of each character, so A is 1 -- into TYPED_LETTERS ($707A), up to the first character below $40, and looked up by MATCH_WORD: the first letter picks a bucket from WORD_INDEX, and each entry in it is unpacked and compared by LETTERS_AGREE over the shorter of the two lengths, until one agrees. The table is each call, with the candidates it unpacked (read at $6F72, where each one starts) and the token it returned (in B and C at $6D34):

Two things happen here that nothing downstream ever sees. A synonym is resolved: SAY's entry ends with a link to TALK, and READ's to EXAMINE, and the token carries the word linked to, so from here on the game only knows TALK and EXAMINE. And the buckets are only grouped by initial letter, not sorted, so a common word can be found late: CURIOUS was the 22nd C-word unpacked.

The same routine, called on single words on a fresh machine, shows its rules for short and long words: a typed word may be shorter than the entry (an abbreviation, taken as soon as the letters agree, so the first in the bucket wins), or longer, but only if the entry has four letters or more and the next candidate does not also agree:

So the tokens for the whole line, as the parser finds them in TOKENS at $6D8A: the class in the top nibble, the dictionary offset from $6000 below. The main loop has made one change: at the closing quote it put a full-stop token first ($B0 $00), so that what is said ends as a sentence.

3. Word classes, and a handler for each

The class is two bits of the entry's first byte and two of its second. CLASS_DISPATCH ($75C1) turns it into an index into PARSER_CLASSES and jumps there; the handlers are the parser. Read from the game (the word counts from the dictionary, synonyms included):

4. The state machine, and the frames it fills

PARSE_COMMAND parses one command and returns to the main loop, which obeys it and calls it again while there is more on the line. It takes tokens through NEXT_TOKEN; register E says what may come next, each handler setting and clearing its bits. A handler that wants to know what follows reads on itself, so not every token goes through the class dispatch: a preposition looks for another preposition, an adjective for the noun after it. The words go into a 24-byte frame at COMMAND_FRAME (more frames, for a sentence that needs them, lie below it, from FRAMES): the verb, an adverb or a direction, and two noun phrases of two prepositions, the noun and two adjectives, each a two-byte word reference, low byte first. Articles are noted and dropped.

Each command below is one call of PARSE_COMMAND: every token read, by which routine (the caller of NEXT_TOKEN), E as it was then, and where the class dispatch sent it; then the frame as OBEY found it at $6D96.

Command 1: TALK TO THORIN, and an order

The frame, as OBEY found it:

The quote is a class $9 token with no word, which PARSE_SPECIAL finds in slot 0 of SPECIAL_WORDS: its handler is SPECIAL_QUOTE. At the opening quote it calls ORDER_BEGINS, which keeps the sentence so far aside and parses what follows in a fresh frame with IS_ORDER set -- QUICKLY, OPEN, THE, CHEST, and the full stop the main loop added, which ends the order as THEN would but, inside quotes, carries straight on. At the closing quote SPECIAL_QUOTE files each frame parsed since into a free slot of ORDERS, marked $FF, puts the first sentence back and goes on. The THEN after it ends the command. So the frame OBEY gets holds only TALK TO THORIN; the order is in ORDERS slot 0, as it was at $6D96:

Command 2: EXAMINE THE CURIOUS MAP

The frame, as OBEY found it:

AND is where the machine looks ahead. PARSE_AND passes over any more ANDs and saves a checkpoint (AND_TOKENS: the place in TOKENS, E, the frame), then ends this part as THEN would and starts a new noun phrase. The next token is DROP, a verb, and PARSE_VERB finds E's bit 3 clear -- just after AND -- so the AND joined two commands, not two noun phrases: it goes back to the checkpoint and ends the command there. DROP IT is left in TOKENS for the next call, after this command has been done.

Command 3: DROP IT

The frame, as OBEY found it:

IT is a special word too. WORD_IT copies the six bytes kept at IT_NAME ($B6E0) -- the name of the last target, set when the command before was matched: CURIOUS MAP -- into the phrase as if they had been typed, and files it. So the frame is exactly what DROP THE CURIOUS MAP would give.

5. From a frame to an action: the sentence patterns

There is no table of verbs. OBEY calls PARSE_ACTION, which calls MATCH_PATTERN: it gathers a probe at $7958 -- the verb, then up to two words picked from the frame's phrases, their prepositions or particles -- and compares it with each 8-byte pattern in ACTION_PATTERNS by NAME_MATCHES, the same comparison the objects' names are matched with, so the order of the two picked words does not matter. The action code is the pattern's place in the list. For the three commands (read at $79C4 and $79E9):

Each pattern's four word references carry four flag bits each, which PATTERN_FLAGS gathers into two bytes, FLAGS_FIRST_WORDS ($B71D) and FLAGS_LAST_WORDS ($B71E). What they mean is read from the code that tests them: in the first, bit 3 the pattern has a target and bit 2 an instrument (PARSE_ACTION and WOULD_WORK), bit 4 it is not narrated (NARRATE_ACTION), bit 5 the target's phrase is the one with the particle (ASSIGN_PHRASES), bit 7 the target is a place and bit 0 PATTERN_OPTION (PATTERN_OPTIONS); in the second, bits 2-3 and 0-1 are the kinds of object the target and the instrument may be (the mode FIND_NAMED_OBJECT searches in: things, characters, or either) and bit 6 the action needs light. Every pattern in the game, with those decoded (the code links to the Actions page, which has each one's handlers and the objects that carry their own):

The ten directions are the patterns with GO as the last word, so NORTH and GO NORTH are one action. Two flags are not accounted for by the table: bit 5 of the second byte, set for GIVE TO alone, which ASSIGN_PHRASES reads when no particle was typed; and bit 7 of the second byte, set for CLIMB OUT OF alone, which NARRATE_ACTION reads to put both particle words before the object.

6. Finding the objects: try, then do

A name is not turned into an object up front. ASSIGN_PHRASES decides which phrase is the target and which the instrument -- by where the preposition the pattern wants turns up, not by order -- and copies their nouns and adjectives to TARGET_NAME and INSTRUMENT_NAME, and the target's to IT_NAME ($B6E0) for IT. Then MATCH_AND_TRY goes through the objects whose name fits, in reach of the actor (FIND_NAMED_OBJECT, IN_REACH), and for each one runs the action through DO_ACTION with DOING_IT ($B6FA) clear: a test. Every handler makes its checks and then calls FOR_REAL, which in a test sets SUCCEEDED ($B6FB) and returns from the handler, so nothing changes and nothing is printed. The first object that would work is kept. OBEY then tests it once more, narrates it (NARRATE_ACTION), sets DOING_IT and runs DO_ACTION for real. For the three commands (fits read at $9E0F, tests at $7AED and $7AF0, the handler at $958C):

Command 1

you talk to thorin.

Command 2

you examine the curious map.
there seem to be some symbols on it but
you cannot read them.

Command 3

you drop the curious map.

The map's EXAMINE is the map's own: its record carries a handler for action 28, and DO_ACTION asks the object before it looks in ACTION_TABLE (see the overview). Here the player is not Elrond, so it only says the symbols cannot be read.

7. The order, on Thorin's turn

TALK TO's handler, DO_TALK, decides how many of the sentences in ORDERS the character takes: a random number up to its limit (6 for Thorin), and ASSIGN_ORDERS gives it that many of those waiting and throws the rest away. Here it was called with 3, and 1 was waiting, so slot 0's first byte became Thorin's number, $3F. On his turn in the world's turn that followed, TAKE_ORDER freed the slot and parsed the frame the way the player's are: pattern 16 (OPEN), target CHEST, which fitted wooden chest; tested, it would work, and he did it:

thorin opens the wooden chest.

The adverb was parsed and kept at offset 2 of the frame, and played no part: MATCH_PATTERN's probe is the verb and the phrases' prepositions, not offset 2 (read), so QUICKLY OPEN THE CHEST is OPEN THE CHEST. How the characters' turns work is on How the characters live.

What is confirmed, and what is not

• Run when this page was built: everything in the tables above -- the candidates, the tokens, the tokens each handler read and where the dispatch sent them, the frames, the order slot, the probes, the patterns and their codes, the objects that fitted, each test's answer, the handlers run, and what was printed. The single-word TOKENISE results likewise.
• Read from the code: what each bit of E means, the pattern flags' meanings (from the instructions that test them; the two single-pattern bits are described by where they are read, not by a run that shows them mattering), and that the adverb takes no part in choosing the action.
• Not worked out: when COPY_VERB_ON moves a frame's phrase to second place, and ALL ... EXCEPT, whose code runs in the build's playthrough but whose result is not checked here.