RetroChallenge 2015/07 Agenda

July RetroChallenge requires moderation for me, so I will keep my challenges simple:

• Retr0bright my recently revived (and upgraded) Macintosh 128K
• Add a few features to my ElWhip web server
• Mess around with an RS232 WIFI device on an old Macintosh
  

Reviving an original Macintosh

About a decade ago, I bought an old, neglected and very yellowed original Macintosh from Craig's List for $20. Inside was a Macintosh Plus logic board, but since the back panel had not been modified, the computer would not even close correctly. The analog board worked, but I wasn't going to modify the case to fit the logic board. I have been waiting for a long time to find a 128K logic board to properly fit inside of it.

I got lucky on ebay a few weeks ago and scored a Macintosh 128K logic board with a RAM and SCSI upgrade, as well as a 800k drive and the Mac Plus bracket for the drive. It was advertised as a "Vintage CPU Design Apple Macintosh 128k to Mac Plus motherboard upgrade kit." $75, including shipping from Canada, seemed like a good deal, since I have struggled to get anybody to let go of one for less than $100.

The board arrived about week ago and had the brand "CPU Design" on it. I screwed everything together, and it booted right up. It had 2.5 MB RAM, SCSI, and newer ROMs. The old Mac finally lives!

Click the link below to see lots of photos.

ElWhip 0.07 with New Features

I added some features to the ElWhip web server, some of which are built in to lwip, and some are completely new:

• Dynamic File Read: The server will read the file in chunks which saves memory and allows much larger files to be in downloaded.
• Dynamic headers: The server generates the headers now and they are not required in the HTML files. This makes it easier to add images and file downloads.
• HEAD HTTP method support: The server responds appropriately to HEAD method requests.
• 'Gentle Quit' in 128K: The application will report an error rather than crashing when trying to quit with 128K RAM.

See the ElWhip Download Page for more details.

RC 2015/01 Epilogue

A member of 68kmla.org was nice enough to test out ElWhip on a real Macintosh 128K: ...and a 512K.

First time getting a REAL Macintosh 128K on the internet. So great!

ElWhip 0.06 Download and Video Finale

ElWhip Webserver 0.06 Download now available.

RetroChallenge 2015/01 Summary:

• Repaired video on my Macintosh
• Compiled and modified lwIP Web server to run on Macintosh
• Wireless logging
• Modified Mini vMac to run virtual serial ports on PPC Mac OS X
• Lubricated all of my old Mac disk drives
• First web server to run on Macintosh 512K
• First PPP connection, TCP/IP stack, and web server to run on a (virtual) Macintosh 128K

So, a ton of work, but a success, even though it could still use a little work to clean up a few things. Thirty-one years after its introduction, the original Macintosh is finally on the web! This was a great experience and I was able to learn a little too much about the Macintosh Toolbox. This was a bigger project than I expected, and it took a lot of tweaking to get everything running in 128K. Thanks for the motivation RetroChallenge!

Loading HTML files in 128K

I spent most of yesterday trying to track down a nasty bug in the lwIP http server. Most of lwIP is very well coded, and other than the CRLF problem I had, very portable and bug-free. The lwIP http server is a bit of a hodgepodge of code though. Lucky me. Anyway, the callback to release the html file resources happens before the server has sent the TCP packet. As a result, I was free()ing memory before the server was done with it which caused corruption in the HTTP headers. I decided to just keep a file buffer up all of the time instead of malloc()ing and free()ing memory as needed. This was a constraint on the size of file that can be loaded though.

I spent the rest of the night moving more PPP code into a separate code segment that I can unload once the connection is up. It worked so well, I was shocked when I saw 9K of free RAM. This easily left me 4K of RAM to load HTML files. I can even open the menus without crashing. Unfortunately, it crashes on Quit on a 128K because it needs too much RAM to bring down the interface.

So, I will get an upload ready and get a video done in the morning!

So close...

So I was able to finish the code to load the HTML files from disk using the original Macintosh File Manager calls. This saved quite a bit of RAM, and allowed be to get the server up. However, now it can't allocate enough memory to actually load a file. At least it logs the error gracefully, rather than crashing: PPP Phase Changed to: 8 App Limit: 91260 Free Memory: 9840 Starting Apps. App Limit: 91260 Free Memory: 2768 Error: "Unable to allocate memory for file." at line 105 in :::ports:mac:include:fsdata_custom.c

I am going to keep working on this today. Tomorrow is the Super Bowl, but I might have time to make a quick video, if I get it working.

Fancy Error Page

I worked much of the weekend trying to get the lwIP web server to load files from a disk drive. This could potentially save some memory because the files could be loaded on the fly, instead of all being loaded into memory at once. I had to enable a lot of additional code to get it working, which more than ate up the memory I hoped to save. Anyway, I reverted all of my code changes, and just fancied up the 404 error page that is compiled into the application. So, my Mini vMac virtual 128K is now running a little web server! I'll try to keep it up through the end of the month.

Macintosh 128K 404 Error Server!

I have been spending a few hours after work slowly trying to conserve memory here and there by delineating parts of the code so that the open and close portions of PPP are in a separate code segment. My thought was I could unload that segment once PPP is up and then load in the HTTP server into the freed memory. This is not working as planned. However, I tried removing most of the HTML pages and the image imbedded in the HTTP server, to see if I was even close on memory usage. I was able to get it serving just a 404 Error HTML Page running in a virtual Mini vMac 128K Macintosh, with just 1K of memory to spare:

404 Error page from virtual Mac 128K. Note Free Memory!
 

Even opening a menu causes a crash.

Memory is so tight that even opening a menu causes it to crash with a 'Memory Full' error code. However, this gives me hope for 2 things:

• If I load the HTML files from disk, as needed, I could probably save some memory.
• Segmenting out some more functions could give me enough free RAM to really get this running.

I will leave it running overnight if people want to try it: virtual Mac 128k 404 Error server. I still need to do some hardware modifications to get my original Macintosh back to 128K RAM. I hope I am getting close...

ElWhip 0.05 Download

I had a major disk drive failure on my 512k. My external 800k Drive doesn't spin anymore, and my other 800k drives aren't compatible with 64k ROMs / HFS. That set my testing back a bit, as I went through and cleaned and re-lubricated all of my old disk drives.

Anyway, ElWhip 0.05 is available for download. Instructions are included.

ElWhip Diet

I have been working on getting my ElWhip web server running in 128K, but I'm not there yet. It is crashing on a 128K Mini vMac right after it negotiates the IP addresses. So close!

The main cause is PPP, which takes up about half of the memory. I tried replacing PPP with SLIP today, but I can't get slattach (SLIP server) to work on Mac OS X 10.4, and it isn't included in newer versions of OSX. So, it's not a good solution. Click for the gory details...

ElWhip Icon

The name of my web server application, "ElWhip" (from 'lwIP'), inspired a Zorro-esqe icon:

'El Whip' would totally be my name if were a Mexican vigilante (or luchador).

It took me a while to remember that I need to set the Bundle Bit ("Has BNDL") in ResEdit for the older versions of the Finder to correctly show the icon.

Web server ran all day today!

Mac 512K on the Internet

I fixed a couple bugs this evening, including the netmask problem I was having. I just needed to set the ppp interface as the default interface ( netif_set_default()) and lwip routes all the packets through the ppp interface, even if its router/ppp server is outside the netmask network.

I enabled ip forwarding on Mac OS X to give public access to my Mac: sudo sysctl -w net.inet.ip.forwarding=1, and set up port forwarding on my router.

And so, here it is, on the internet for the first time: my Macintosh 512K Web Server Running System 2.0 and Finder 1.1g (http://72.220.233.84:8080). It is just using the default webpage right now - I need to set up a better one. It seems to work from my iPhone. It is 30 years behind on security updates, so please be kind to the old thing!

Leave a comment if it works for you. I'll try to get a video up this weekend with more details.

CRLF ≠ LFCR

I thought it would be an easy and quick project for last night to go back and properly fix the CRLF problem I was having. Unfortunately, it was not. I spent hours and hours testing the HTTP input string, hex-dumping, debugging, testing, and comparing strings using the lwip's built-in data compare functions. I still could not find what was going wrong. I started losing faith in the lwip code, the MPW complier, the Motorola 68000 processor, all of computer science. Was this really happening? It is getting so late, am I dreaming?

Finally, I double-checked a hex dump on the string to search for: "\r\n" and it reported: 0x0a 0x0d. At first, I thought 'that looks right', but then I noticed that it should be returning 0x0d 0x0a instead! How could it be transposing these predefined character escape sequences? A quick search on google revealed the problem: MPW transposes these two standard escape characters by default. I could't believe it. Of course, it is documented in the Apple SC/SCpp Reference for the MPW C/C++ compiler:

The compiler will by default map \n to the value 10 and \r to the value 13 to conform to the MPW environment. The -noMapCR option will turn off the mapping of the newline to carriage-return and carriage-return to newline.

If you are familiar with Macintosh line breaks (CR only), this makes sense, but if you are familiar with ANSI C code, it does not.

In the context of the lwIP web server, it was erroneously looking for 2 LFCRs, but wouldn't get them until there were 3 CRLFs: CR[LFCRLFCR]LF. Once I regained my sanity, the fix was easy:
/* #define CRLF "\r\n" */ /* WTF? I hate computers... */ #define CRLF "\x0d\x0a"

Compiler character mapping quirks, swapped characters and pulling hair out all seem to be themes in the RetroChallenge.

Debugging Payoff

My new debugging setup, along with increasing the logging levels in lwip, quickly revealed some of the problems I am having. I managed to tackle a few of them last night.

First, I noticed my lwip ppp interface is not attempting to re-negotiate a connection. So, if it fails the first time, you have to quit the program and start again. This isn't very user-friendly, so I added a timed function to check the ppp status and, if necessary, alert the user and restart the ppp negotiation. This makes recovery from a timing problem much easier.

Next, during a failed connection to the webserver, I would get a warning like: pppos_input[0]: Dropping bad fcs 0xc540 proto=0x21
Basically, lwip was dropping ppp packets. The bad fcs was misleading and caused me some headaches. Finally, I noticed that every time this happened, about 4 lines before the packet was dropped would be a log entry: pppos_input[0]: got 63 bytes Now, lwip ppp reports got xx bytes whenever it gets data from the serial port. It is the 63 bytes that is significant, and was associated with the dropped packet. I happened to retain some information from my book, Inside Macintosh, Vol. II:

Each input driver's buffer can initially hold up to 64 characters, but your application can specify a larger buffer if necessary.

Ah! So, lwip was getting the opening part of a packet, which was cutoff by the buffer overrun, then getting another packet, ending with the checksum. The checksum wouldn't add up, so it dumped the packet (actually two packets). I think it reports 63 bytes rather than 64 bytes because the buffer is a pascal string with the first byte representing the string length(?). Fortunately, Inside Macintosh, Vol. II also provided a solution:

SerSetBuf allows you to specify a new input buffer, replacing the driver's 64-character default buffer.

So, I added a 256 byte buffer and the connection is much more reliable. It even significantly improved the rate of initial ppp negotiation.

Debugging the Debugging

I have been working on a way to get the proper debug logs out of ElWhip without using the modal dialog boxes which cause the program to stall until a user presses the 'OK' button. I have decided to just dump the logging output through the printer port, since I already have serial port code for ppp and the modem port. Unfortunately, that means I would need another serial connection between the Macintoshes. But, feeling inspired by another Retrochallenge project, I set up a bluetooth rs232 adapter for a wireless connection. I initially had a problem where bringing up the serial connection to the printer port would bring down the connection to the modem port, but I worked through a couple of bugs to get it working.

This should make it much easier to troubleshoot some of the intermittent problems I have had establishing the ppp connection and retrieving the web page.

 Wireless Logging with Bolutek RS232 - Bluetooth Adapter 

Virtual Distractions

I had a major distraction last night that kept me up way too late. One of the major limitations of the Classic Macintosh emulator Mini vMac is the lack of serial port realization (is that the right word?). There is a hack at spritesmods.com that includes a virtual tty connection from Mini vMac. The code is for an older version of Mini vMac and has a few bugs that prevent it from working on Mac OS X. This would be a perfect testing platform for ElWhip, if I could get the serial ports working. A lot of troubleshooting got me to here:

Mac Terminal in Mini vMac with tty to host computer

Unfortunately, it won't drain the incoming data until I press a key, so it is often displaying the input a little bit behind the output (eg: you have to start typing the username before the prompt appears). I think it needs a polled function to check for available data. I don't think it will work with ppp as-is. Hopefully, I can figure it out tonight.

Update: That was easy, I re-implemented an unused SCC_Update() function in the main run loop and it works great! Now to test ElWhip on it!

Update: Mind blown:

Elwhip running in Mini vMac with ppp connection to host

No web page though. Going to bed before I create some sort of causality loop. On that note, weirdest thing about Mini vMac: its build environment is a Mac 68k application that runs within Mini vMac.

Uptime On Old Time Macs

Too many chores to do today, so I just let the web server run on the Macintosh SE, checking to see how stable it is. Running ping -i 60 192.168.11.130 | cat -n from the Mac OS X terminal pings every minute and lets me know how many minutes the TCP/IP stack stays up. It started losing pings around 65 minutes and eventually timed out altogether. This seemed to correspond with the After Dark screensaver on the Mac SE. I turned off the screen saver (and turned down the brightness knob), and the server ran for 174 minutes before I turned it off. There were with no dropped pings and it was still serving the web page when I stopped it.

After that, I ran the web server on the original Macintosh (still with 512K RAM), and had similar success. As far as I know, this is the first Macintosh web server with 64K ROMs or 512K RAM, ever! Certainly the first on System 2.0 and Finder 1.1g.

Web Serving! - barely

I can serve a web page from the Macintosh SE! There was a problem where lwIP does not seem to be detecting the first CRLF (Character Return + Line Feed) required by the HTTP spec. So, when I used Telnet, I had to hit return 3 times before the HTML appeared. Web browsers will only send 2 CRLFs, then wait. I ended up deleting one of the checks for a CRLF from the code, and it worked! However, that doesn't explain why it isn't detecting the first CRLF. I think it could losing the last character of the buffer (LF) or something.

The process of dealing with this made me realize I can't keep overlooking a few problems I have seen:

• Getting PPP to connect is too timing-sensitive and flakey
• I can't always get the web browser to connect
• PPP is a memory hog
• I need to actually fix all of the "FIX ME"s I have been inserting into the code
• I need a better logging / debugging system

Some of these problems are related. I need a good debugging system system before I can fix any of them. Enabling debugging currently causes a bunch of modal dialog boxes to appear. This causes further timing problems and does not allow logging for analysis.

Once I get some of these issues sorted out, I will test it out in 128K and make a download available.

HAKKO, how have I lived without you?!

Repaired Solder Joints

MacMemory Address Decoder Board

My Hakko Soldering station arrived today, and although I had intended on waiting until this weekend to do more RetroChallenge labor, new tools make it hard to wait. Anyway, this thing is amazing. It melted the solder joints so fast, I couldn't believe it. I am used to using dollar store soldering pens - NEVER AGAIN!

My Mac liked it too! Video came back on boot.

In other news, I took a close look at the RAM expansion circuit that was added to my Macintosh to address the additional RAM. I think I can just connect riser Pin 1 and 2 to make it a 128K Macintosh again for testing.

Edit: Actually, I will remove the 47 Ohm resistor too, just in case.

Macintosh Webserver Recap and Restart

lwIP is a lightweight TCP/IP stack that was designed for embedded systems with, or without an Operating System. During RetroChallenge 2014WW, I was able to get MPW 3.5 compiling programs for old Macintosh System versions, port lwIP, and get a TCP/IP stack running on my Macintosh with 512K RAM. For RetroChallenge 2014SC, I was able to update my working version of lwIP and compile the webserver before life interfered.

---

I woke up this morning without much of a hangover, allowing for an early start on the challenge.

I updated my local copy of the lwIP source which has now incorporated the ppp-new branch into the main branch. It is nice to see continued development on lwIP. This updated ppp code was important because it allowed for ppp without multithreading, which is necessary for the early versions of Macintosh OS. With git installed, two quick commands did the trick: /usr/local/bin/git clone git://git.savannah.nongnu.org/lwip/lwip-contrib.git /usr/local/bin/git clone git://git.sv.gnu.org/lwip.git

After copying over my additions, and converting the end of line characters to be Mac friendly, I was able to compile it with MPW again.

I fixed the error about the fsdata source file not needed for link. As it turns out, that file is included through one of the header files, and does not need to be compiled separately as a source file.

Finally, I transferred the program to my Macintosh SE for testing (thanks ZIP drive). The PPP and TCP/IP connections come up fairly reliably, but, I couldn't connect to the webpage using Safari.

I Telnetted to port 80:
PowerMac-G4:/Volumes/Mac OS 9/Maclwip151 epooch$ telnet 192.168.11.130 80 Trying 192.168.11.130... Connected to 192.168.11.130. Escape character is '^]'. GET / HTTP/1.0 404 File not found Server: lwIP/1.3.1 (http://savannah.nongnu.org/projects/lwip) Content-type: text/html <html> <head><title>lwIP - A Lightweight TCP/IP Stack</title></head> [...]
And it seems to have worked! I will have to keep testing this evening and look through the code a bit to see why it is not loading with Safari (or FireFox).

I am already further along than RetroChallenge 2014SC!