Address space Limitations of the AMD64/Intel64

Started by PabloMack, July 31, 2019, 03:45:01 pm

Previous topic - Next topic

PabloMack

August 03, 2019, 09:11:14 pm #15 Last Edit: August 03, 2019, 09:13:07 pm by PabloMack
Quote from: WASasquatch on August 03, 2019, 03:45:31 pmAnd how does this relate to AMD64 2GB limit of address space?

Are you familiar with WOW, what it means and what it is for? (and I don't mean World of Warcraft)

WAS

Quote from: PabloMack on August 03, 2019, 09:11:14 pm
Quote from: WASasquatch on August 03, 2019, 03:45:31 pmAnd how does this relate to AMD64 2GB limit of address space?

Are you familiar with WOW, what it means and what it is for? (and I don't mean World of Warcraft)


You mean WoW64? I don't know of any WOW. I don't follow. WoW64 is layer for 32bit processes on a 64bit OS.
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

PabloMack

August 04, 2019, 08:31:07 am #17 Last Edit: August 04, 2019, 10:55:06 am by PabloMack
Quote from: WASasquatch on August 03, 2019, 10:01:04 pmYou mean WoW64? I don't know of any WOW. I don't follow. WoW64 is layer for 32bit processes on a 64bit OS.

Yes. That's what I mean. Thank you for the correction. This is probably something you already understand. It is a way for a 64-bit OS to provide an environment that appears to be identical to the only one that a 32-bit OS can provide for a program to run. A 32-bit process can't see outside of this address space because that is all there is. The supervisor that manages such a process, though, runs in a 64b-bit mode so that, from the perspective of the running process, it appears to be the same as though it were using a 32-bit OS. 

When the same program is compiled and linked to run in a 64-bit address space, it's segment bases can place it anywhere within a much larger address space. According to the video that I posted earlier about OpenVMS, only 48-bits of the 64-bits are supported. But this is still a much larger address space than the 32-bit environment provides. And even though it is a 64-bit process, the instruction set of the AMD64 is still limited to the same set of addressing modes though there are a very few more. But what is available to one that is running in 64-bit mode is not enough to make it a full 64-bit program within a 64-bit process. The standard addressing modes available to any program using the AMD64 instruction set still limits it to only see within the 32-bit window where it resides. It is like taking your house that is on the ground floor, and hoisting it up into a high-rise so that it is turned into a condominium. Many of such houses can now share a large address space. When you walk around your own house, it is still the same size, even though it is no longer sitting on the ground by itself. But just because your house is sharing a larger address space with other houses, it doesn't make your house any larger on its own. Any one person within his house still only has the same reach when using the addressing that can be used by the linker. Beyond this, though, there are a few computational tools to reach anywhere within the full 64-bit address space. But this must be done after it is linked and loaded (i.e. during run-time). It would be like a guy who can go visit someone else's condominium by using the stair wells. When he does, he can no longer reach the things that are in his house because he didn't grow longer arms. But he can now reach the things in the neighbor's house. With a true 64-bit program, the guy would be able to reach the kitchen cupboards in his neighbor's house to borrow a cup of sugar without leaving his own condominium. The whole condo high-rise is like the 64-bit process, but the size of his house and length of his arms are like the 32-bit program running inside it.

So I guess I am saying that, on an AMD64/Intel64 machine, there is no such thing as a 64-bit program, only 32-bit programs that run as part of a 64-bit process. Can you accept that statement?

WAS

August 04, 2019, 12:42:29 pm #18 Last Edit: August 04, 2019, 12:45:20 pm by WASasquatch
I'm still confused. Your explanations seem reasonable, but they fall apart outside the forum, where none of the issues are even recorded. I still cannot find anything on this limitation, and a single 64bit process (program) can address 4GB of address space (btw, which is virtual before physical why it's crucial to really match your physical despite arguments here) on AMD64/Intel64. This is just how it is, so there must be something you're not understanding correctly.

Again, the LAA hack for even 32bit programs, allows this same SINGLE process use of up to 4GB addressable VIRTUAL memory (RAM cache is different).

The only thing I found regarding a limitation on size due to "48bits" is PAE and 32bit processes. PAE is riddled with problems and should be avoided and buried.
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

PabloMack

August 04, 2019, 02:49:28 pm #19 Last Edit: August 04, 2019, 02:51:56 pm by PabloMack
WASasquatch, I mean no disrespect. If you were an avid AMD64 programmer at the assembly level, you could look at what I showed you in the AMD64 specification and you would easily come to the same conclusion I did. It is like a mathematician who doesn't need to ask someone else for an equation or find it in a table in a book somewhere. He knows enough to derive the equation from scratch by knowing the rules. Almost nothing happens in Windows or Linux without the CPU executing instructions. And so it is by understanding the instructions and their addressing modes you can surmise their limitations. I doubt that many people on gaming forums understand enough to do this for themselves. They need to search the web to see how many people conclude about some issue by consensus. This is not understanding.

WAS

Quote from: PabloMack on August 04, 2019, 02:49:28 pmWASasquatch, I mean no disrespect. If you were an avid AMD64 programmer at the assembly level, you could look at what I showed you in the AMD64 specification and you would easily come to the same conclusion I did. It is like a mathematician who doesn't need to ask someone else for an equation or find it in a table in a book somewhere. He knows enough to derive the equation from scratch by knowing the rules. Almost nothing happens in Windows or Linux without the CPU executing instructions. And so it is by understanding the instructions and their addressing modes you can surmise their limitations. I doubt that many people on gaming forums understand enough to do this for themselves. They need to search the web to see how many people conclude about some issue by consensus. This is not understanding.


I mean no disrespect either, and I think you're misunderstanding the specifications in some fashion. As this isn't discussed ANYWHERE else, and you'd think programmers would be talking about it when building their software to universally run on 64bit [Windows] systems.

Beyond you now indexed on Google, I can't find anything from developers.
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

WAS

August 04, 2019, 03:03:53 pm #21 Last Edit: August 04, 2019, 03:07:15 pm by WASasquatch
Which brings us back to your original post and Terragen and stuff, and how it really relates.

Also still confused by calling AMD64 64bit really 64 when it's kinda common knowledge AMD64 is a true specification over Intel, where it's an adaptation spec.
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

PabloMack

August 04, 2019, 03:34:56 pm #22 Last Edit: August 04, 2019, 03:36:51 pm by PabloMack
Quote from: WASasquatch on August 04, 2019, 03:03:53 pmAlso still confused by calling AMD64 64bit really 64 when it's kinda common knowledge AMD64 is a true specification over Intel, where it's an adaptation spec.

Can you rephrase this for me? It is not clear what you mean.

WAS

August 04, 2019, 03:41:12 pm #23 Last Edit: August 04, 2019, 03:43:48 pm by WASasquatch
You said AMD64 is not really 64bit, but a hack, but AMD64, through R&D has always been considered a true implementation of 64bit specification, where Intel modifiying NetBurst as an adaptation to the market. For example like I mentioned early PAE with 64bit for 32bit processes is a mess from Intel.

Additionally, the specifications you linked in Volume 3 shows 3 dedicated 64bit ops? with the exception of the others.
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

WAS

I'm curious, you're a assembly developer? Couldn't you just demonstrate this 2GB limitation with some 4GB asset in a single process calling for 4GB call and attempting to cache it safely in RAM?
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

PabloMack

August 04, 2019, 03:58:12 pm #25 Last Edit: August 04, 2019, 04:03:51 pm by PabloMack
I never used the word "hack". It's not like me.

Take a look at the following which is taken directly out of microsoft's PE32/PE32+ specification. They extended PE32 so that programs could be run in a 64-bit address space. Notice that on the line "SizeOfCode", the size is 4 and is unchanged from PE32. So tell me how many bits is in 4 bytes and what the address range (i.e. "limit") would be for an offset of this size.

WAS

August 04, 2019, 04:20:07 pm #26 Last Edit: August 04, 2019, 04:25:24 pm by WASasquatch
I'll reiterate some past posts.

  • What is the relation to your proposed limitation and Terragen (While TG does use .NET, I'm not sure what, so not sure the 2800mb+ memory issues apply). And like Matt said, the code space must be small
  • Can you demonstrate this limitation, as I cannot find anything on it. Specifications don't mean anything to me really, and again, as far as programming for 64bit, doesn't relate. 4GB calls ARE made in programs, on both AMD64 and Inte64 by a SINGLE process.

For example, like I mentioned before, with Morrowind, LAA extension allows 4K gameplay with hundreds of mods, making calls above 2GB which is the vanilla limitation. So if these calls were limited to 2GB on a single process, it wouldn't really matter and Morrowind would be fine with all mods and calls going through virtual memory -- but if you run Morrowind without the executable modified with the flag, it will crash with due to calls larger than it can handle.

Which is a great example of the same process running, even 32bit, Large Address Aware, able to bypass this as a single process.
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

PabloMack

August 04, 2019, 04:24:22 pm #27 Last Edit: August 04, 2019, 04:26:31 pm by PabloMack
Just answer my questions and think for yourself.

1. How many bits is in 4 bytes?
2. What would be the address range (i.e. "limit") for an offset of this size?


WAS

August 04, 2019, 04:29:48 pm #28 Last Edit: August 04, 2019, 04:32:55 pm by WASasquatch
Quote from: PabloMack on August 04, 2019, 04:24:22 pmJust answer my questions and think for yourself.

1. How many bits is in 4 bytes?
2. What would be the address range (i.e. "limit") for an offset of this size?



Again, I'm not really interested in specifications you may be interpreting poorly, and that mean nothing to me, but actual working code (literally for over a decade in a lot of instances) that makes these single process calls, and these single process calls even documented in specification too.

You came into the topic entirely misunderstanding Windows Task Manager, for one, which is very disconcerting, and have yet to even explain the ramifications clearly, let alone demonstrate it as a assembly programmer (?). Plenty you can use to demonstrate it, and can easily fabricate a dummy asset to use with Windows/Linux in a single command. I'm sure you could practically demonstrate a real world 2GB limitation over the established 3GB/4GB extensions (and prove it without a doubt).

At this point I'm not really even interested in the conversation.
Check out the Terragen Community on Facebook: https://www.facebook.com/groups/Terragen.Galleries/

PabloMack

August 04, 2019, 04:57:15 pm #29 Last Edit: August 05, 2019, 01:30:55 pm by PabloMack
I don't know why you won't answer two very simple questions so I'm not interested in the conversation either.

What makes a programmer and an end user fundamentally different is that a programmer has to be able to understand what does not yet exist. In order to do this, he has to understand what the capabilities are for the tools he is going to use to build it. So specs are a lot more important to me than to you because without them, I am blind and what I want to make will never be made. I can't just try this and try that. I will never get very far if I depend on that. As an end user, seeing a working program is believing. The proof that something can be done is seeing someone who has already done it. But you will not understand how it was done so you just have to guess, take someone's word for it or try to get a consensus among all of the end user observers.  That kind of evidence is worth very little to me so that's why I have just brushed it off.

In your Terragen work, you don't just copy what someone else has done. You want to make images of things that have never been seen before. At least you can see that much.

On a positive note, I have learned a couple of things in all of this. In the past I have used Task Manager mostly for killing programs that were hung in infinite loops and for other reasons like to see what is running. But now I realize that the memory (Private Working Set) means that it accounts for all of the memory used by the process, not just the image. I would say that "entirely misunderstanding Windows Task Manager" is an overstatement.

After I provided you with "evidence" that you asked for and you then said you were not interested in it. It took me a while to realize that evidence was not what you wanted. You just wanted corroboration and I can't corroborate my own findings. We do define our words differently.