Virtual Reality for Void Expanse! Custom Oculus Touch Mapping!

[ninekorn]

Here's the Google Drive link to a video of the project! It took me 1 month and a half to code a lot of stuff Inside of the program. I am using the AB4D DXEngine Oculus Wrap and the Ab4D Oculus sample and the SharpDX libraries! The video is 500mb long approx 26min. Sorry for the bad language in the video if there is any.

Enjoy watching.

https://drive.google.com/open?id=1m38AZpK5U77MSCW6ysHYfev6Vn3l_S62

ninekorn

[Lurler]

This looks quite cool actually, but I imagine it doesn't really add anything to the game, or does it?
But either way, awesome!

[ninekorn]

This looks quite cool actually, but I imagine it doesn't really add anything to the game, or does it?
But either way, awesome!

Thank you Lurler 

No, it doesn't change the gameplay and it probably never will.

BUT, I am onto something there, that might be a way to change the appearance of the background if the background's color is constant in ARGB. I did some tests with a constant background color, but unfortunately for the moment, the performance is not high enough and my processor cannot handle dealing with  8 294 400  bytes every frames (which is 1920p multiplied by 1080p multiplied by 4 because every pixel in the screen is 4 bytes), in fact, it could only deal with 1/4 of the screen pixels and NOT every frames and so it was useless and ugly to look at for the moment. Basically, I can loop through the bytes array that contain the image to find the bytes that correspond to the "Constant Color" and then change the colors programatically. My goal was to make the color with an alpha value of 0 (i think 0 is for transparent) so that I can add a 3d background. I will need to work more on this though, as I can't promise it will ever work.

Void Expanse already has a 3d background and although we can see the effects, it still remains a 2d image that we look at. But If I make the background invisible in Virtual Reality, I can put ANYTHING at ANY depth behind the Virtual Screen that won't Interact with the game Itself. It would just make the game very beautiful to Watch in Virtual reality with a "real" 3d background.

Also, I "might" just be able to fiddle a little bit with a "mesh" and change the depth of the vertices of the mesh to match "ships" and "stations" in the game. BUT that, I didn't try it yet because I am working this week and I am K.O. when I arrive home 

[ninekorn]

Just wanted to clarify that it never is going to change the gameplay... Only the background visual and maybe foreground. I am not into "hacking" void expanse . I succeeded in making the background completely transparent in virtual reality so we see only "decorations/objects/ships/stations" but I am somewhat lacking knowledge in speeding things up with c# pointers. I am working on that right now... I just wish to be able to do another tutorial this weekend but I am caught up in making this work.

I am trying MemoryStreams and Pointers (which I have a big gap knowledge in accessing bytes per bytes for MemoryStreams)... Right now I am at 2 milliseconds overhead in just making the player area transparent. I can't seem to bring it down to 0 but at 1920*1080 it still is 736000 bytes to check every frames for the player ship area that starts at 720 in the X and 360 in the Y coordinates.

I have been 1 week on this junk and I am starting to lose confidence.

By the way, if any Void Expanse players have an Oculus Rift, please write here. I'd love to know how many we are.

Here's a code snippet that I can't make any better for the moment.

Code Select Expand

  fixed (byte* textureByteArray = _textureByteArray, previousTextureByteArray = _previousTextureByteArray, currentByteArray = _currentByteArray)
            {
                byte* buffer0;
                byte* buffer1;
                byte* buffer2;

                for (int x = xxPlayerShip; x < xxPlayerShip + widthOfRectanglePlayerShip; x++)
                {
                    for (int y = yyPlayerShip; y < yyPlayerShip + (heightOfRectanglePlayerShip); y++)
                    {
                        bytePos = ((y * 1920) + x) * 4;

                        buffer0 = &textureByteArray[bytePos + 0];
                        buffer1 = &textureByteArray[bytePos + 1];
                        buffer2 = &textureByteArray[bytePos + 2];

                        if (*buffer0 + *buffer1 + *buffer1 >= 750)
                        {
                            byte* buffer00 = &currentByteArray[bytePos + 0];
                            byte* buffer11 = &currentByteArray[bytePos + 1];
                            byte* buffer22 = &currentByteArray[bytePos + 2];
                            byte* buffer33 = &currentByteArray[bytePos + 2];


                            *buffer00 = 0;
                            *buffer11 = 0;
                            *buffer22 = 0;
                            *buffer33 = 0;
                          }
                     }
                }
           }



[ai_enabled]

@ninekorn,
usually any stuff which is dealing with framebuffer data should operate as a GPU shader, otherwise you're forcing GPU to flush all the graphics command to access the framebuffer data, then modify it and push back. No wonder it's ultra expensive.

With a GPU shader - you can call it a GPU post-processing shader - you can perform the same operation per pixel very effectively. And the code of such shader will be super simple as you basically need only the framebuffer for input and a very short pixel shader to modify the color of each pixel.

Unfortunately, I don't have any knowledge of how you can hook into the graphics API with Oculus, but there are should be definitely a pretty simple way - as you're already doing something very similar, but in a different way (by copying the framebuffer from GPU to RAM and modifying it with CPU, and then pushing it back to GPU).

Regards!

[ninekorn]

@ninekorn,
usually any stuff which is dealing with framebuffer data should operate as a GPU shader, otherwise you're forcing GPU to flush all the graphics command to access the framebuffer data, then modify it and push back. No wonder it's ultra expensive.

With a GPU shader - you can call it a GPU post-processing shader - you can perform the same operation per pixel very effectively. And the code of such shader will be super simple as you basically need only the framebuffer for input and a very short pixel shader to modify the color of each pixel.

Unfortunately, I don't have any knowledge of how you can hook into the graphics API with Oculus, but there are should be definitely a pretty simple way - as you're already doing something very similar, but in a different way (by copying the framebuffer from GPU to RAM and modifying it with CPU, and then pushing it back to GPU).

Regards!

I am using the byte array created by the screencapture from XooF (Alexandre Mutel). Once I get this byte array, I am creating a ShaderResourceView from the Ab4D Engine (Ab3d.DirectX.TextureLoader.CreateShaderResourceView) and then I assign that to the texture material shader. That was the fastest way in the Ab4D engine. It's super fast. But computing the white background of 1920 * 1080 * 4bytes = 8294400 bytes is crazy expensive.

But I succeeded... 10 minutes ago... hell yeah , well at least with 736000 bytes for the moment out of 8294400 bytes. This is ultra fast. I am down to 0 milliseconds. And of course nothing interferes with any game whatsoever. I am really pleased with the results. I'm gonna try to map the rest of the 1920*1080 screen and see where it goes

[ninekorn]

nevermind I think I fail.

[ai_enabled]

@ninekorn, if you can use ShaderResourceView (Direct3D 11, right?) to render the texture (taken from the framebuffer), then you can also write an HLSL effect (shader) for this ShaderResourceView which will do the required operation per pixel. And it will be almost instant as the job done only on GPU.

And if you can totally avoid the byte arrays and can directly apply the shader to the frame buffer it will be the best solution performance-wise. It should be possible to avoid ScreenCapture and simply hook to the application itself.

[ninekorn]

Thank you for your advice... I have just backuped this project and I am rerolling to one of my earlier version at least to make the basic stuff working for the VR desktop. Andrej Benedik from Ab4d engine also proposed to me a similar approach where I could achieve at least 10 times the speed in the screencapture, before incorporating the transparent background for Void Expanse. I will start working on that after the basics. I know nothing about shaders either so I might leave all of those for next weekend.

There's still some stuff that don't work in the basics anyway, like the Windows Keyboard that is displayed on screen at the press of one Oculus Touch button but I can't even click on it as it seems to be on a layer of it's own.

Gosh, my best shot at the transparent background was only getting an area of 460 x 400 x 4 pixels around the player ship covering the zoom on the ship and already there it was jungling between 0 and 1 milliseconds. By cutting all the screen of Void Expanse UI bitmap from the screencapture to a byte array and applying transparent colors, I would probably reach 4-5 milliseconds and that would have been noticeable in game which is not really pleasant for players.

[ai_enabled]

@ninekorn, honestly, I don't feel anyone will ever play like that as in the result it's still far from what most players might consider as a good experience.
But as an experimental project to further improve your programming skills it's a great idea.

Don't be afraid of shaders. They're basically a single file with a pretty simple code - all it does, usually, is just a pixel color modification based on all the inputs (such as textures and texture coordinates). Some shaders are indeed quite complex, but writing simple stuff is really easy.
You can learn something from CryoFall. For example, the Lighting/Combine shader applied every frame after the sprites rendering is finished to make the picture transparent ("lit") in the lit places and dark in the non-lit places - it receives a combined "mask" texture of all the light blobs and simply combines it with the previously rendered framebuffer.
You can find the code here <CryoFall folder>\Core\Core.cpk\Content\Effects\Lighting file Compose.fx. You can modify the result to return, for example, only green color - simulating something like a night vision: (in the end of the MainPS function)

Code Select Expand


result.rb = 0; // reset red and blue components of the pixel color
return result;

(the game will automatically instantly rebuild the shader so you can do live-coding and play around; just be sure that you've enabled Developer Mode in the game options to display any shader compilation errors you might have (as otherwise, you will never know that there are any errors))

Imagine how much more code will be required to code something similar by using ScreenCapture and pointers - not mentioning the prohibiting performance overhead. The shader is executed almost for free even on my 4K (3840x2160 pixels) screen - almost 0 ms on CPU and less than 0.5 ms on GPU, it seems.

Of course, HLSL (or GLSL, for OpenGL) is a completely different language but a very simple one.

You can also find a lot of great examples with open source code on http://shadertoy.com (just don't be spooked, many shaders presented here are a piece of art and really complex for understanding even to a professional Visual FX programmer - simply because mostly they're coded in an obscure way that only their creators can understand what they're doing :-D ).

Regards!

[ninekorn]

 

I'll be sure to check the shader reference website that you pointed out.

 

Lol, but I don't agree that players won't like the Virtual Reality experience once I get everything I want done for Void Expanse. I don't want it to just be a Virtual Screen for players to look at without anything else involved. I would rather NOT have had to explain parts of my complete project  but here it goes:

• In the "Invasion" Mod, players will still be in the Solar System Inside of a ship and the only way to get to Void Expanse Galaxy system, will be by Long Range communications system (not literally of course). A little bit like the current nowadays drone technology where army personal are behind a screen controlling a drone. Players won't "BE" there in the Void Expanse Galaxy, but what they will control are Hybrid/Robot/Android like characters that can control the ships there in the Void Expanse Galaxy.
• This is where the screen Capture and VR comes into play when players in VR sit down in front of a computer that is inside of a ship and login to the Void Expanse Long Range Communications System to control their ships there in the Void Expanse Galaxy in front of a 2d/3d like screen in Virtual Reality. Lol I know right, a little bit far fetched considering only the Screen Capture is done, the galaxy market isn't completed, the weapons and items generator isn't completed, the follower and drones mod isn't completed etc...
• I want the experience to be a one of it's own but although I have been a couple months in the AB4D engine, it's a totally new experience to me so I am pretty much starting from the bottom.

I need a LOT of stuff to be working in order to complete this Overhaul/mod/Proj3Kt.
Right now, there is only ONE way that I can get the complete list of coordinates of systems/decorations/objects/ships/stars/etc without asking for more API functions that would write them down in an XML file or something... It is part of a personal project too. AI "Image/Objet/Writing" Recognition. As long as I can debug the total coordinates of those objects in the Void Expanse Console, I can definitely code an external AI script to read what is written on the Void Expanse console without Interfering with the game... (not with my current knowledge and the fact that I don't have the time to learn that for the moment but I did find some really nice references)

But that brings issues... Like only the creator of the Game Server would have access to those coordinates unless my external VR application can dispatch the coordinates to Players computers that aren't creating the server.

If I can get the transparent background working at some point, I know that I will be able to synchronously make players see 3d objects in the transparent background that they would normally see in the normal Desktop Void Expanse. But again that will bring other issues as those objects aren't transparent in the normal Void Expanse Universe. I was thinking of making a planet/decorations etc. 3d objects microscopic so that I can swap them for 3d objects in Virtual reality that are visible in the background.

It's gonna take me 100 years to code all of that... Or will it? Lol... so far fetched. I should start finishing my Void Expanse mods first. I am happy thought that the only game that I am modding is Void Expanse so everyday that I code is not for another game.

I want to code all of that for Void Expanse also for the reason that at some point later on, I will be able to "port" what I have coded for "real life" RC drones. Lol... dudes I know I feel crazy right now but I assure you that I'm not lol

[ai_enabled]

@ninekorn,
sorry for the late reply. We're super busy with CryoFall A15 update.

The scale of your ideas is truly grand. With the right direction, you can indeed achieve a lot. But you definitely need to start with something smaller and more practical and finish it.
When I'm saying more practical, I mean not only that it should be truly useful for a lot of people, but also that you should have proper means of implementing it. Hacking around is not practical as you're very limited by scripting API and available system/DirectX hacks to implement what you're looking for. Unless you're a masochist :-) , it would be better to make your own game - make a simple VR game on a simple but interesting idea, and complete it (and put it on itch.io or even try to sale on Steam). So much stuff to explore there. And your creativity will be not limited by scripting API or anything else. Especially if you're using Unity as it has a lot of stuff already integrated - great for creative minds and VR enthusiasts (if you had any problems with VR here before, most likely they're resolved - it's actively used by about (or maybe over) a million of developers now).

Regards!

[ninekorn]

@ai_enabled

Well my VR project has been put on the shelf anyway for now - I've received a reply from the AB4D creator and there is a lot of space for improvements which I would have loved to implement yesterday as per his suggestions but dust is going to pile up on it for sure. Here's the reason why:

1. one month ago approx, my Oculus Rift VR sound started to stop working intermittently. Now I have no sound but the image still works fine. It's been reported by many many users on the web that the ribbon cables that go from the Headset straight to the headset earplugs have a manufacturer flaw. The more often that you "stretch" the headset head frame to put it over your head, the more the ribbon cables get stretched and they end up breaking. I did not open the headset but I know for sure that this is the same problem. Less than 1 month warranty and I hate using warranties.

2. My processor AMD FX8320e 8 cores just broke yesterday at the same time as
3. The motherboard MSI 990FXA-GD65-V2 but it's not all
4. The Video card NVIDIA EVGA 960GTX Superclocked (already RMA'ed once in 3 years because it broke) also broke down.

Dudes... How the heck can that happen. 3 pieces of hardware in one shot and it wasn't even overclocked and everytime the CPU and VideoCard were overclocked, they always reached under a maximum of 55 degree Celsius at full load. What's even worse is that the FX8320e 8 cores warranty ended at the beginning of this year, the motherboard MSI 990FXA-GD65-V2 warranty ended at the beginning of this year and the Video card NVIDIA EVGA 960GTX Superclocked ended at the beginning of this year.

But I must admit that my terrain generation tryouts with using multiple threads to do so for at least 4-5 months might be one of the problems root. And I have been working for the past 2 months on VR non-stop with a video card 960GTX that is under the minimum specs requirements for the Oculus Rift.

BUT, yesterday, on spare parts, I built a working system:

MSI 760GM-P23(FX) motherboard
AMD Phenom II x4 925 2.8ghz (oced to 3.4ghz with a LEPA watercooler - with that it's still less than half what my FX8320e could push)
MSI RADEON R7 240 video card
and obviously the rest of pieces that didn't break/burn yesterday.

I am obliged to leave VR alone cos even if using the Oculus Debug Tool to reduce the Oculus Rift Resolution of default 1.00 to 0.25 (1/4 resolution), it is awfully lagging. I can't even do squat haha.

The good news is I can now concentrate on Void Expanse and push my mods and tutorials but lost 1 day out of my weekend to do so. What a week of crap lol

Let me work today on at least trying to release something lol

[ai_enabled]

@ninekorn,
wow, that's sad to hear!
Usually, if you have such a widespread issue, it might be a problem with the PSU. You can't say for sure if some component is broken unless it's obvious (burnt, etc) or you have tried it in another (compatible) system. Try to get a new PSU or ask at the local service center if they can do the diagnostics. Maybe CPU and MB are ok and you will need to get a PSU replacement.
I believe Oculus will replace the cables as it's definitely a manufacturer's defect.

[ninekorn]

@ai_enabled

Thank you for the hints.

Yesterday when I tested all the stuff on the micro-atx motherboard MSI 760GM-p23(fx), I had put only 1 bar of ram and in the wrong slot so no wonder why nothing was working. But the PSU was working because I've been using it on that new build.

Another good news in all of this is my NVIDIA EVGA 960gtx is actually working with the re-testing today.

I installed the latest version of BIOS on the micro-atx motherboard MSI 760GM-p23(fx) which was supposed to have compatibility with the AMD fx-8320e processor but unfortunately, it seems to be dead.

So maybe my motherboard MSI 990-GDA65-V2 might be working after all but I wont swap all my stuff again just to test it.

Anyways. I'm tired of not releasing anything so I'm gonna work hard on 1 mod to start off and see where it goes from there.