A note about performance

While the new shader generates quite nice results, we also want the program to run smooth. Using 20 number of steps for both linear and binary search gives us steady 60 FPS on my old desktop with nVidia GTX 460 and Intel Q9550 @ 3.9 ghz (2560x1440 resolution). Increasing to 30 binary search steps immediately results in FPS drops down to 40 FPS. It's really hard to spot any difference between using 30 and 20 steps for binary search, there is a bigger visual difference between 20 and 10. 20 steps seems like a good number.

Some more comparisions

Some more pictures comparing the new shader, blur and no blur, with the old parallax diffuse shader.

No blur:

Blur:

Parallax Diffuse:

  

Looking at these, I really think the second one with blur is the best. It isn't as harsh as the one with no blur. Also, it's arguable if the height is enough. Adding more height would make a more poping effect, but I do need to have in mind that bricks really don't have some much of a pop effect.    

This is a comparison using the blurred height map with different heights:

Same height as above:

Slightly more height:

There is no big difference here, as I only added a very small amount of height, but the difference could really make an indirect impact on the user even if the user doesn't really see a direct difference.

Though, this is a secondary question. The most important thing in this project is that the shader can do both, then it's up to then end user of the shader to decide what is best for the particular scene. 

The new shader with new height map!

I've worked on a proper height map today, using gimp adding a transparent layer to the original texture. I tried a couple of different levels of gaussian blur, for smoothing out the edges a bit. I ended up using some blur and the result were pretty sweet.

This is a comparison between the to height maps (with and without blur) in action:

No blur

Blur:

As you can see, blur really smooth things out. Though, while you don't want the bricks to be rounded, you don't want them to have those artifacts they get without blur either. It's certainly something that could be worked on to get the best possible result.

This is really a huge difference compared to the old parallax diffuse shader:



Though, this pictures doesn't make justice. In Unity, the difference is even greater, especially comparing the colors.

These are the height maps used:

No blur:

Blur:

The specific RM shader.

I spent the whole afternoon yesterday writing the shader for this specific project, trying to make it as minimalistic as possible. The result became pretty good, the shader does everything I want it to do, and nothing more. 

Implementation in CG:

Defining the variables that should be used in this shader:

  sampler2D _MainTex;
  sampler2D _NormalMap;
  sampler2D _HeightMap;
  float _Height;
  float _Gloss;
  float _ColorIntensity;
  float _Specular;
  
  struct Input { // vertex input
   
   float2 uv_MainTex;
   float2 uv_NormalMap;
   float2 uv_HeightMap;
   float3 viewDir;

  };

I think each of them is pretty self-explained.  The struct represent the incoming parameters; uv coordinates of the textures applied and view direction.

Luckily, writing shaders in CG for Unity doesn't involve that much math. The ray is set up:

  IN.viewDir = normalize(IN.viewDir); 
   
  // set up the ray
  float3 p = float3(IN.uv_MainTex,0);
  float3 v = normalize(IN.viewDir*-1);
  v.z = abs(v.z);
  v.xy *= _Height;

Implementing the ray tracer:

  // ray tracer with linear and binary search
  const int linearSearchSteps = 20;
  const int binarySearchSteps = 20;
   
  v /= v.z * linearSearchSteps;
   
  int i;
  for( i=0;i(lessthen)linearSearchSteps;i++ )
  {
      float tex = tex2D(_HeightMap, p.xy).a;
      if (p.z(lessthen)tex) p+=v;
  }
   
  for( i=0;i(lessthen)binarySearchSteps;i++ )
  {
      v *= 0.5;
      float tex = tex2D(_HeightMap, p.xy).a;
      if (p.z(lessthen)tex) p += v; else p -= v;
  }

This the exact same procedure that was explained in the post about Relief Mapping. As you can see in the binary search, we move up if the height map shows a greater value. If it shows a smaller value, we move down.
Note: The html editor didn't want me to use > and < signs.

Next thing is to generate the output:

  // generate the output  
  half4 tex = tex2D(_MainTex, p.xy);
   
  half3 normal = UnpackNormal(tex2D(_NormalMap,p.xy)); // normal map

  normal.z = sqrt(1.0 - dot(normal.xy,  normal.xy));
  OUT.Normal = normal; 
     
  OUT.Gloss = tex.a*_Gloss;
  OUT.Specular = _Specular;
  OUT.Albedo = tex.rgb *_ColorIntensity;

 The _Gloss, _Specular and _ColorIntensity variables manipulates the output of the texture. Which is basically created out of the mods I did to the test shader.

The last thing is to implement what parameters that should be modified from the Unity interface. In addition to the most obvious (textures, specular color, height), I added those variables showed above.

  
  _ColorIntensity ("Color Intensity", Range(0.5, 1.5)) = 1
  _SpecColor ("Specular Color", Color) = (0.5, 0.5, 0.5, 1)
  _Gloss ("Gloss", Range(0.5, 1.5)) = 1
  _Height ("Height", Range(-0.05, 0.05)) = -0.01
  _Specular ("Shininess", Range (0.01, 0.1)) = 0.014
  _MainTex ("Base (RGB), Spec (A)", 2D) = "white" {}
  _NormalMap ("Normalmap", 2D) = "bump" {}
  _HeightMap ("Height (A)", 2D) = "bump" {} 

This is about it for the implementation. A very minimal relief mapping shader, that works flawless for its purpose. When I've created a proper height map, I'm gonna post some result using this shader and the new height map. 

Testing

Currently, I'm using an existing relief shader that i found. It's fairly simple and pretty much what I need for figuring out what I actually need in my shader, that is to be implemented soon enough. I've also made a testing height map, which is so bad that I won't show it here... :)

However, I'm done some testing and here are some comparisons:

Parallax Diffuse, view 1:

Unmodified test shader, view 1:

As you can see, the test shader adds alot more depth to the surface, but the height map isn't that good, so some parts looks better then others. 

Some of the reddish colors I talked about in the last post, is created by the parallax shader, seen while comparing the two. Though, it's still to reddish when using the test shader.

I decided to take a look inside the test shader, and ended up modifying some attributes. Also, I cranked up the binary search steps a bit, though this change doesn't do so much until I have made a proper height map. The result was way better:

Parallax Diffuse, view 2:

Modified test shader, view 2:

The modified attributes were simply a quick fix. I added a multiplier of 0.9 to the gloss attribute and a multiplier of 0.8 to the rgb color scheme. For the binary search, I added 5 more steps, which is a whole lot. But it certainly made some difference.

As the testing has given me some clues about how to implement the actual KTH brick wall relief mapping shader, I'll begin implementing tomorrow or something. 

Note: The shader used for testing can be found here: http://forum.unity3d.com/threads/fabio-policarpo-relief-mapping-with-correct-silhouettes.32451/

Project: Some more details.

I currently working on the relief mapping shader for the KTH main building bricks, the goal is to make it look as much realistic as possible. The current shader which is used since the creation of the model last year is a parallax diffuse shader.

The stuff that I'm working with (creds to the guys that created this last year):

Original brick wall texture

Brick wall normal map:

KTH admin building with parallax diffuse shader:

The first thing that hit me was that the colors wasn't so realistic looking. The overall reddish is much more dull in reality. This becomes more clear when zooming in a where the light intensity is pretty high:

Though, this will be fairly easy to fix when implementing the new shader.

Theory: Relief Mapping

It's time to check out the actual relief mapping technique, as we now know how parallax occlusion mapping works, this will be pretty straight forward.

In parallax occlusion mapping the illusion of depth was created by stepping fixed values, which gives a pretty nice result, but not as nice as we want it to be. What we want to do with relief mapping is to find the exact displacement for every fragment. In other words: the surfaces depth will be an exact match with the values that the height map gives us. If we can achieve that, then we could generate a very accurate illusion of depth in our surfaces.
How can we be so exact without making the ray tracer run forever? We use binary search. Say, we needed a step size of 1048576 to get a good result with our ray tracer, that isn't really practical. With binary search that could be achieved with only 20 steps (2^20 = 2048). Binary search in the ray tracer works like this: The currentHeight is initially set to 0.5, then we step either up or down. That depends on what our height shows: if the height map value is greater then our currentHeight we step up with currentHeight/2. If it is smaller, we step down with currentHeight/2 and stores this as a possible match. This procedure continues for the specified number of binary search steps.

Now, we got a match, but it isn't really exact yet. If an object occludes another object, using binary search, we will not get the correct silhouettes. It's pretty easy to understand because of what we stated above: we start height / 2, we step up, finds an intersection. Then we never will be stepping down, but the correct match could still be down there, because of this occlusion.
This can easily be solved though. If we use a simple linear search with a big step size before our binary search, we can find an intersection which is fairly good. This intersection is then refined with the binary search, just searching in the appropriate section, and there we have our exact match.

As you easily could understand, this shader is quite more hardware consuming then the parallax occlusion mapping shader. But when finding a sweet spot with the binary search step size and the linear search step size, we could use this in real time applications. On fairly powerful machines, this shader would run with pretty good frames per second. These days, it would be quite possible to have it as an enthusiast option in video games. As Intel Skylake, AMD Zen and the Radeon 300 series is upcoming, I'm sure we will see some popular game engines implementing this in the future.

Thats about it for the relief mapping shader. Gonna post some details and updates about the project next.  

  A pretty cool result of using relief mapping with shadowing that I found.

Theory: Parallax Occlusion Mapping

The easiest way to understand the relief mapping completely is to first understand the parallax occlusion mapping shader. The parallax occlusion mapping is quite similar to the technique we will use, but a bit easier to start with. This is probably be the toughest part theoretically, relief mapping should be pretty easy to understand when this part is clear.

Intro

Parallax occlusion mapping is a popular technique used in many real time applications and modern video games such as Grand Theft Auto V, The Elder Scrolls: Skyrim and Crysis, where the goal is to make realistic looking textures. Parallax occlusion encodes the surface information in textures to reduce its geometric model's complexity while giving a fairly realistic look. To manipulate the surface details a height map is used, which represents displacement in the texture or surface. A height map is often based on the original texture but in gray scale, where the difference between black and white represent the material's displacement. More about height maps later. The texture's details are reconstructed in the pixel shader, using the height map, when the model is rendered, so that it creates an illusion of displacement (in most cases depth).

 A good example of a parallax occlusion shader applied in GTA V. As you can see, the stone bricks really seems to have depth, but it's really just a 2D texture.

Have in mind though, that you need a bunch of other techniques to generate something like that, like anti-aliasing for example. That said, this would look totally boring without the parallax occlusion applied. 

  

The algorithm

We know that the idea is, with the help of a height map, to generate volumetric shapes for each pixel of the rendered surface. This sounds pretty advanced, but with some basic linear algebra math it's not that hard. Before we get into the algorithm itself, I'll show a illustration about what we want to with it and how our height map works. 

As I stated before, the height map is based on the original texture, but covered in gray scale. If we say that the scale is ranging from 0.0 to 1.0, where 0.0 is black and 1.0 is white, we could describe the height in the texture. The more precise the height map are, the more precise will the illusion of depth in the surface be. 

 Example of a well done height map for some brick wall texture.

Back to the algorithm - I hope my poorly made illustration that follows, is enough to get the basic idea of how parallax occlusion works.

What you see here, is the actual ray trace in the pixel shader that we want to implement. The thin lines under the flat surface represents the height map values, and the red line represents the ray that's being traced. The point 1 is where the ray first intersects with the surface, this point represents the texture coordinates we would normally render. Instead, we wait until the ray intersects with the height map. When it does, we can calculate our illusionary coordinates, which gets rendered instead. Lets move on to some more technical (linear algebra) stuff! 

The main loop of this shader is going to find the intersection of the camera vector with the height map. To skip some extra processing, we are going to quit the loop as soon as any intersection is found. 

The vertex shader

Here is what's going to happen in the vertex shader:

1. Calculate the vector from the camera to the vertex. 

1.1. Transform the vertex position into world space and subtract its position from the camera position. 

1.2. Subtract the world space vertex position from the light position to find the light direction vector.

2. Transform camera vector, light direction vector and vertex normal to tangent space.

2.1. Create the transformation matrix with the binormal, tangent vectors and vertex normal together with the world matrix. Be sure to take the transpose of the matrix as we want to transform from world to tangent and not from tangent to world.

2.2. Use the transformation matrix just created to transform the vectors to tangent space.

2.3. Multiply the incoming vertex position with the world view projection matrix to get the correct output position.

The pixel shader

This is where the actual parallax occlusion mapping takes place.

1. Calculate the parallax constants.

1.1  Calculate the maximum parallax offset and it's direction vector with the help of the texture coordinate, the value our height map is showing and the tangent vector from the pixel to the camera. The camera vector must be normalized to give the offset direction vector.

1.2. Determine the number of samples that should be used. 

1.3. Calculate the step size. This is simply done by divide the maximum height (1.0 as stated earlier) with the number of samples. 

2. Set up the core.

2.1. Initialize the variables used in the main loop: currentRayHeight, currentOffset, lastOffset, lastSampledHeight. currentSampledHeight, currentSample. 

2.2. Create the main loop. The main loop should run while the current sample is less then the total number of samples.

2.2.1. Calculate currentSampledHeight.

2.2.2. Check if currectSampledHeight is bigger then currentHeightRayHeight. 

If it is, set values:

2.2.2.1. Calculate delta of currentSampledHeight and currentRayHeight, also the delta of currentRayHeight plus the step size and lastSampledHeight. 

2.2.2.2. Calculate the ratio which should be applied to lastOffset and currentOffset by dividing the first delta with the first delta plus the second delta.

2.2.2.3. Calculate currentOffset with the ratio applied to lastOffset and currentOffset. (ratio * lastOffset + (maxHeight - ratio) * current offset). 

2.2.2.4 Set currentSample to maximum number of samples to quit the loop.

If not, step:

2.2.3.1. Decrement currentRayHeight by step size, increment currentSample by one.

2.2.3.2. Set lastOffset = currentOffset and add (stepSize * maxOffset) to currentOffset.

2.2.3.3 Set lastSampledHeight = currentSampledHeight.

3. Set the finals.

3.1. Set the final coordinates to texture coordinates + currentOffset.

3.2. Calculate the final normal and the final color with the help of the different maps used. 

3.3 Optional: Manipulate the outgoing color and other parameters. 

--

Thats pretty much it. The tricky part is in the pixel shader, but these steps together with a linear algebra book should make it doable.

Overview

As I stated before, this project is about creating a relief mapping shader for the KTH simulation brick wall textures. A relief mapping shader is, basically, a shader that creates the illusion of depth in a texture. The relief mapping shader uses ray tracing with height maps to calculate how the depth illusion should be applied. The use of ray tracing together with binary search makes the result very appealing, in most cases a lot more accurate then when using some other depth illusion shader technique, such as parallax occlusion mapping or parallax diffuse mapping. This technique though, is quite dependent of the use of a modern machine. Ray tracing with binary search is not an easy task even for the modern computer, when posting the theoretical part about the relief mapping technique I'm going to describe some methods to make this process a lot quicker while still generating great results. Next post is going to be quite heavy theoretically, describing the algorithm itself and how it works compared to the parallax occlusion shader. Maybe I'll split it up into two parts, we'll see about that.

A pretty nice result of relief mapping:

The Project

This project is about creating and applying a relief mapping shader to a brick texture used for the KTH simulation project. I'm going to write both theoretical and more demonstrational posts.