No it's definitely the alpha channel. I tried compressed and uncompressed on my PC at work with the same result. All of my greyscale masks are LZW compressed and I haven't had any problems at all with images up to 11,000x11,000 (although my normal maximum is 8,000 x 8,000).
For the repeat test I just created a new document in Photoshop to make sure it wasn't something from WM. They will work OK if you either remove the alpha channel *or* convert them to RGB, so it's something specific to the combination of greyscale with alpha channel.