Ticklish Techs

I am playing around with R#-plugins these days. I have started off with this nice introduction. Unfortunately it silences about some problems that may occur to you in the beginning.

1) When trying to copy the given source code you’ll mention that all the classes are unknown to Visual Studio, because all a lot of references are missing. So far I have no clue about the R#-class-architecture. So I started .NET Reflector, open ALL the .dlls in the R#/bin-folder into Reflector and use the search to find the .dlls to reference in my project.

2) When you have your first R#-plugin ready you want to test it. To do so you have to copy the dll containing your plugin in the R#-plugin folder. You may want to do this in a post-build event like this:


copy $(TargetPath) "C:\Program Files (x86)\JetBrains\ReSharper\v5.1\Bin\PlugIns\WobTest"


When you build your project with this post-build event it fails. This doesn’t work for two reasons. First: Win 7 denies access to that folder. Second: after another start of Visual Studio R# has loaded the plugin so you can’t copy a newer version over it.

My solution to this is a .cmd that starts Visual Studio without R# (devenv /safemode). I run this .cmd as administrator.

Now at least I can debug my plugins by starting another instance of Visual Studio (not in safemode!)

This solution is not cool, because developing R#-plugin without using R# in the development-process sucks…. any ideas?

a better solution…

Some minutes after I published this post, Benjamin pointed me to this page. Forget about my lines to 2). You don’t need a silly .cmd run as administrator not a post-build event. All you need to do is to start your debugging Visual Studio the the command line option /ReSharper.Plugin followed by the path to your plugin-dll.

/ReSharper.Plugin "R:\sandbox\ReSharperPlugInTest\PublicGoesVirtual\PublicGoesVirtual\bin\Debug\PublicGoesVirtual.dll"

When I get a new laptop that always triggers a little cascade in my home network. I have three machines: An older laptop serves as a server, a quadcore desktop pc provides gaming power and the a newer laptop accompanies me wherever I go.

Some day ago I replaced the old server laptop (with WinXP) by a newer one, now running Win7.  My printer (Brother DCP-120C) is a little aged now and it’s not a network printer. So I attached it to the server (via USB) and thus it’s accessible in the entire network.

Attaching the printer to Win7 worked flawlessly. Importing the printer (which involves copying the drivers) to another Win7-machine was not a problem either.

But my quadcore still runs WinXP and wouldn’t accept the drivers from the Win7-server. Instead a dialog asks me to provide the appropriate driver manually.

Okay, so I downloaded the latest DCP-120C-drivers for WinXP, unzipped them and guided XP to the .inf-files. XP said: “No, I can’t see the right drivers.”

Unfortunately there is no way to force WinXP to use certain printer-drivers. So I spend ten minutes playing the old yes-no-game before I took a look at the .inf-file. I have never dealt with .inf-files a lot but I noticed this line:

"Brother DCP-120C Printer"   = BRDP120C.PPD, BrotherDCP-120C65FA

Hmm… okay… it is there. My Brother DCP-120C Printer. Fine. Why doesn’t WinXP recognize it? I found another .inf-file in the driver-package with these line:

"Brother DCP-120C USB Printer"   = BRDP120C.PPD…

Hmm… now the name is DCP-120C USB Printer. Do the funny name string make any dfference? I took a closer look at the printer device that was recognized by my server:

Okay, nice, there is no “Printer” in the name… can it be… well… I copied the line in the .inf-file and replaced “Printer” by “USB”:

"Brother DCP-120C Printer"   = BRDP120C.PPD, BrotherDCP-120C65FA
"Brother DCP-120C USB"   = BRDP120C.PPD, BrotherDCP-120C65FA

Bingo! That did the trick.

When accessing printers installed on other network computers, the printer drivers really seem to be recognized by their name. That hurts a bit, but knowing it may help solving driver problem from time to time.

Whenever I move my system to a new laptop I create a virtual machine from my old system. Sometimes it is easier to access stuff I forget to copy from a running system than from mounted backup devices.

To create virtual machines from a running system I use VMWare Workstation, which outperforms VirtualPC by far (imho).

Some days ago I tried to create a VM from a Win7/64 system. I was quiet surprised when VMWare Workstation told me, I needed Administrator rights:

So what – I have administrative rights. But the Workstation didn’t believe me. Since that held me from moving to my new laptop I was really p*ssed.

Luckily some smart guy found this solution.

I think this is clearly a bug, but so far VmWare Workstation demand Administrator-rights in the literal sense: It only accepts, when you are logged in as THE Administrator:

• start c:\Windows\system32\cmd.exe from explorer using "execute as administrator".
• type net user Administrator /active:yes
• logoff & reboot
• login as Administrator
• create your image
• Complain at VMWare.

I am a great fan of Acronis True Image. I am using it for a long time and it has saved my skin more than once. So I was glad to hear that the new version (True Image 2010) was designed to work with Win 7. And so it did.

Last week I got a new laptop and today I tried to backup my fresh installation ov Win7/64. Each time True Image caused my entire system to freeze after five seconds of backing up. No reaction at all, the only way out way a hard button-press reset. I tried different options without results. (Which made me panic, because on what else can you rely when not on your backup tool?) –

Fortunately I found the solution here.

There is a new driver for hdd-image snapshots. Installed it, True Image 2010 works again.

Phew!

Some time ago we blogged about a nifty little .net assembly that enables you to access the basic Windows shell operations in shell32.dll . You may want to do so especially in Win7 because your file operations integrate with the fancy Win7 dialogs, the flashing progress bar etc.

The problem we blogged about in the former post was a missing "Pack"-instruction in the marshalling-description of a structure. I guess we didn’t test it with WinXP64… but some days ago we tests it with Win7/64 and it failed. We figured out that we introduced a new bug that occurred on 64 bit machines only. After some more testing we found out that:

• The original code (without Pack-value) crashes on 32 bit machines (as described in the former post).
• The original code (without Pack-value) works fine on 64 bit machines (but we don’t think the original developer had 64 bits 7 years ago 😉 ).
• Our fixed code (with Pack=2) works fine on 32 bit machines.
• Our fixed code (with Pack=2) crashes on 64 bit machines.

It seems that the "SHFILEOPSTRUCT" structure in shell32.dll has different layouts in Win7/64 and Win7/32. I guess the guys at MS aim for performance and tell the compiler to optimize. The compiler optimizes by assuming the optimal Pack-value, which usually is the byte-width of the processor; 4 for 32bit, 8 for 64bit.

What effect does the Pack-value have at all?

The "Pack"-value controls the alignment of the starting addresses of each single member of a structure (often called "offset").

Let’s say you have this struct:

Usually you don’t care for the exact way your data is stored in memory, but when you pass it from .NET to the Win-API you have to make sure Win-API receives and returns exactly the right format. This process is called "Marshalling".

One crucial instruction for doing so is the StructLayout attribute.

(Note: There are a lot more options for the StructLayout attribute and a lot more attributes that help you at marshalling.)

LayoutKind.Sequential means that your data is represented in memory in the same order as you declared it: First "sweet16", than "bite" and then "tanga".

Pack=1 tells .NET that every byte can be the starting point of the next member. So .NET produces this structure in memory:

Now Pack=2 makes sure that only every 2nd byte can be the starting point of the next member. That leads – of course – to unused bytes in memory:

The higher the Pack-value, the more unused bytes you get:

Is this a waste of memory? Sure. But processors can access those addresses a bit faster than other addresses. So it’s an performance/memory tradeoff.

After some more testing, I figured out that the correct Pack-value for Win7/64 is 8, while the correct Pack-value for Win7/32 is 2 (see former post). Why 2? I’d expect 4 here. I don’t know and Microsoft wouldn’t provide the source code I guess.
Maybe in former Windows version they preferred a middle-course between performance and saving memory.

The simple solution

Now it was obvious how to make die ShellLib-assembly work with 32 and 64bit: Check what machine we run on and use a different marshalling. Since I had to declare different structs here, the major code is copying data back and forth, not very interesting.

The only interesting part here is: How do you check on what kind of machine you run? I was looking for some property at System.Environment, but didn’t find anything. Then Andreas pointed me to this post and the solution is too simple to cross one’s mind:

“Just do a IntPtr.Size (static method) and since IntPtr is designed to be an integer whose size is platform specific, it will be 8 if you are on a 64-bit machine and 4 if you are on a 32-bit machine.“

You can find more information and our fixed code soon at the codeproject.com .

Epilog

You can also do the marshalling all by yourself, starting with memory allocation and ending with freeing memory, just like in the old times. But this is another story and will be told later.

Maybe.

Here is a nice little challenge, Benjamin came up with some months ago: “From a list of numbers find the smallest and the largest one – but do not use more than 1,5 comparisons per number.”

Dear reader: Try it!

The solution is simple, straight forward and very nice. And I didn’t find it.

I found something else. My approach was to look at the largest-number-so-far (max) and the smallest-number-so-far (min) as the boundaries of a region. A number outside this region must be larger than max or smaller than min and causes that boundary to change.

To check if a number (a) was outside that region a little computation came to my mind:

check = (max - a)*(min - a);

You would expect n to be smaller than max, so (max-n) should be positive.

You would expect n to be larger than min, so (n-min) should be positive, too.

Two positive numbers multiplied result in a positive number again.

So, if n was outside the boundaries, check would become negative. I only had to check which boundary was exceeded and that’s it:

if (check < 0)
    if (a > max)
        max = a;
    else
        min = a;

Using this approach the number of comparisons  converges to 1 when the length of the list grows. So I found a solution that is way better than 1,5 comparisons per number, didn’t I?

Boooooh

No, I didn’t. I cheated. In fact  (max-a) and (min-a) are comparisons, they just don’t use > or <.

(Actually it’s the other way around: To compute < or > most processors do a subtraction and compare the result to 0.)

So – if you count the subtractions as well you get 3+ comparisons per number…

The intended solution to the challenge (and the code you should provide in your exam) is:

a = list[count++];
b = list[count++];

if (a<b)
{
    if (a<min) min = a;
    if (b>max) max = b;
}
else
{
    if (b < min) min = b;
    if (a > max) max = a;                    
}

So – what’s the point of this post?

The point is: My silly approach can be faster than the standard solution. Depending on the type and range of the numbers in the list calculating and probing check needs less time than the comparisons in the standard solution.

It may not be much, but sometimes small advantages matter.  For example: For an integer-list of length 10^8 with values from -10.000 to 10.000 my approach is 0.02 seconds faster (on my current laptop). And has less code.

So if you feel like using it – I won’t charge you.

Ten days ago I promised a proof why the differences of any permutation of the same digits is always a multiple of 9. Here it is

Let’s say x and y are digits and ‘xy’ is not ‘x*y’, but the number that consist of the digits x and y.

xy – yx = multiple of 9.

Is it always true? Yes, it is:

(10x + 1y) – (10y + 1x) = multiple of 9?
9x – 9y = multiple of 9?
9(x-y) = multiple of 9? Yes, for sure.

I calculated the numeric values of xy and yx by "10 times the higher position + 1 time the lower position", just like we all do every day in our beloved decimal-system. More formally we describe the factor for the position (1, 10, 100…) by

10^position, (position is zero-based, counting from right to left, of course)

Now, when we change the position of a single digit within a number we change its numeric value from

x*10^(old position) to x*10^(new position).

We can neglect x here, because it’s a factor that occurs in both values, so we can get rid of it by division.
The remaining part for building the difference is

10^p1 – 10^p2, a formula that always produces multiples of 9: 10-1, 1000-100, 10-1000000.

Conclusion

Since the difference of a number and one of it’s permutations is a sum of (10^p1-10^p2)-parts, which are all multiples of 9, the total also can be divided by 9.

Lately I stumbled over permutations and noticed a funny fact: When you take any integer, produce a permutation and subtract one form the other, you always get a multiple of 9.

It works with a any length. Some examples:

Length 2
23 – 32 = -9 
84 – 48 = 36
60 – 06 = 54

Length 3
123 – 132 = -9
and so on…

It still works for changing the front position:
123 – 213 = -90

In fact it works for changing any position:
123 – 321 = -198 (-22*9)

This leads to the conclusion I stated in the beginning. The difference of any permutation of the same number is a multiple of 9. Feel free to check it with you favorite numbers.

It’s not a great mathematical invention (if it is, please let me know), but I didn’t know it, didn’t expect it und find it kind of fascinating. I asked a couple of friends and no one was aware of this.

I still try to find a way to use these fact, maybe for spell-check-like for numbers or a fast calculation of permutations, but so far it seems just useless 🙂

Instead I can provide a proof for numbers of any length and I’ll post it here in ten days from today. Meanwhile I want you to try it yourself. The proof is really, really simple, almost trivial and it won’t cost you more than five minutes to understand it all.

Please email or comment your solution!

For those who look for a quick solution:

Try switching off the win32manifest-switch in Visual Studio.

Here is the long story.

Hi! As you may already know we are working on the best .NET – ARIS interface there is so far. It’s an interface that makes heavy use of an old C-Api, using interop and p/invoke.

In Mid-December Benjamin found a very strange behavior that almost spoiled my Christmas. We had just release Version 1.1 when he called me: “Arisan crashes with Vista64.” I was shocked. “And with Vista32 as well”. I felt annihilated. We had done so much testing with various versions of XP, Vista and it worked fine with any of those OSes.

After some testing I saw that ARIS didn’t deliver any pointers at all. That’s rather bad for an interface to C that uses pointer in every call.  After one more day we had more evidence: We had switched to .NET3.5 and VS2008 some month ago and our old versions, that were compiled using the .NET2.0-compiler, still worked.

I was so terrified. The 3.5-framework should be similar to the 2.0-framework, just some (not so) little enhancements. A bug in the 3.5-compiler?

To provide a quick solution we re-wrote parts of the fancy 3.5-syntax, to make our newest enhancement compile with the 2.0-compiler again. That worked (at least we thought so) and we released version 1.1.1.

Then a very frustrating time began. We read articles over and over and quickly found this very interesting post. The problems described there where exactly what we saw.  So we followed the track of the nxcompat-flag and DEP, but it didn’t lead us anywhere.

The C-Api we interface is well documented, but it calls hundreds of other components and one of them crashed deep inside an undiscovered country.

We read pages of differences between 2.0- and 3.5-compiler, new compiler-flags (should have paid more attention here), any documentation we could find, but nothing seemed to help.

We had a feeling it could have something to do with Vista UAC, but we didn’t find any matching explanations to our problem. Being really desperate we started this thread at the MS forums. As you can read there, no one could provide a solution that worked.

I was really sad, because the last thing I wanted to do was to switch the entire development of arisan back to VS2005. I got prepared for a very sad life from now on.

But five days ago my life turned to happiness again. One more time I started of by googling words like “.NET compiler 2.0 3.5 differences” and one more time I read the new features of the 3.5-compiler. This time I obviously paid more attention to the compiler-flags and read about the /win32manifest and /nowin32manifest – switches. I didn’t have much hope left, but tried to set the /nowin32manifest switch.

It worked! Everything worked! The C-Api delivered pointers again. I tried switching the win32manifest on and off and a day later Benjamin confirmed my results. We had found it. But after a long time of suffering (for me it felt like half a year) we still were skeptical and thought we needed a deeper understanding of the problem.

The win32manifest and UAC

Finally we had something to look for and quickly found valuable information on the great I’m just sayin’ blog. The documentation of the switches helped as well, so here is a summary:

The win32manifest is not the .NET-manifest.

The win32manifest is part of the UAC (user access control). UAC was introduced with Vista as an answer to more and more security threads. One of its ideas is to keep malicious software from secretly starting to work. If you work with Vista you may have noticed a lot of dialogs that ask you for permission to run certain programs. THAT is UAC.

Now the win32manifest inside an application tells the OS what level of permission it needs to run. There are levels like “asInvoker” (default), “highestAvailable” or “requireAdministrator”. In case the OS (and the current user) can provide the requested permission, the program runs as a trusted process (according to the granted access level). Of course, 90+% (just my estimation) of the programs on you Vista-machine don’t have a win32mianifest,  because it’s a new feature and older applications just don’t have it. (And I think a majority of newer applications won’t have it either). In order to be compatible with no-win32manifest applications on one hand and providing security on the other hand, Vista can do a miracle called “virtualization”. It’s a bit like running a process in a sandbox: Vista provides anything the process may need to run: disk space, memory, a registry etc. So the process feels cosy.

But anything provided this way is separated from the main system. The process uses a copy of the registry, writes to a special parts of the HDD drives and accesses the memory in a controlled way (not sure about the memory thing…). For me this is a great achievement of Vista, almost magic. I can’t tell in detail how it works (mainly because I don’t have a clue), but it works good and doesn’t cost notable performance.

So on your Vista machine you have processes that run virtualized and others that run non-virtualized. “Non-virtualized” is also called “UCA-compatible” or just “compatible”, so I’ll adopt this wording here.

You could say that Vista trusts compatible processes a little bit more than virtualized processes. That doesn’t mean virtualized processes are evil, but.. well… maybe… under certain circumstances… not 100% trustworthy. (Here is another nice article)

Note three facts here:

1. The state (virtualized/compatible) always counts for a process and is set when the process starts. When you start an .exe-file, Vista decides how to run it.
2. Because of 1) there is no sense in attaching a win32manifest to components that do not start a process, like .dlls. Thus .dlls don’t have a win32manifest.
3. The state (virtualized/compatible) of a process doesn’t change during its lifetime.

So if a .dll is used by a virtualized process, its code also runs virtualized. And if the same .dll is used by a compatible process, its code runs compatible.

Interaction

With some of that in my mind, I could explain the problems we had with our interface software arisan: arisan is a .dll you reference in your .NET-application and use it. As mentioned above arisan calls a native (so non-.NET) .dll using p/invoke-interop-techniques. Now this native .dll starts a bunch of new processes: a database-server, a database-driver, some middle-layers, etc. I am pretty sure non of this newly started processes is based on applications with a win32manifest – so they all run virtualized.

But the .NET-applications we use to test the arisan.dll started running compatible as we switched to VS2008 and the 3.5-compiler. So our test application (compatible) called arisan.dll (compatible), arisan.dll (compatible) called C-Api.dll (compatible). C-Api.dll started new processes (virtualized !!) and called functions there. And in the end a compatible (fully trusted) process asked a virtualized (less trusted) process to fill some pointers… and that’s the point where Vista interferes and says: “Dudes, are you drunk? No way!”

It was a bit of bad luck that the clash of a virtualized and a compatible process happened so deep inside a software we couldn’t debug and only threw some strange, meaningless error-messages, but on the other hand Vista could have told us more. Maybe there is some place in Vista where the clash is logged. If someone knows – please let me know, too. Let me get this straight: Vista is absolutely right here, but I’d wish more information when it happens. I don’t exactly know how, but you could do harmful things if you could easily exchange all sorts of pointers between virtualized and compatible processes…

Solution

So finally the solution was easy. Both processes have either to run virtualized or non-virtualized. Since we can’t make a third-party software run compatible, we have to make our .NET-application run virtualized: We just need to prevent .NET from inserting a win32manifest to executables.

Using the compiler from the command-line the compiler-flag /nowin32manifest does it.

In VS2008 same switch can be found on the “Application”-tab of the project properties.

The manifest-area is enabled for ‘console applications’ and ‘Windows applications’. It is disabled for class libraries.

Another way to handle this problem is to switch off UAC completely. But we strictly advice you against this solution. UAC is one of Vista’s main features and it is designed to provide shelter for your precious data in times of evil viruses and brutal intrusion attempts 🙂

Epilog- bad fix

After we understood the problem, we saw that the ‘fix’ we provided as V1.1.1 couldn’t do much. Ok – it made the demos run, because the executable compiled with the 2.0- come without a win32manifest and thus run virtualized. But users trying it with VS2008 still had the same issues.

Actually we cannot provide a technical solution with the arisan.dll, because if the running mode depends on the executables that use arisan. So we’ll write detailed information and everyone lived happily ever after.

I love CodeVeil. From my point of view CodeVeil is simply the best tool to protect .NET assemblies from being reflectored, decompiled or whatever. It is very powerful, offers a wide range of protection aspects and still comes with a handy gui. So far I haven’t seen a method to crack veiled assemblies and that’s why I really rely on it for my commercial products.

You don’t find bugs in CodeVeil when you try to find them, but half a year ago we found this one by chance. CodeVeil veils well, but the resulting code throws a mighty exception.

You don’t need spectacular code to lure the bug out of it’s hole – here is the prelude:

private static void Main(string[] args)
{
    // Get list and fill it with values
    List<Capsule<int>> l = new List<Capsule<int>>();
    l.Add(new Capsule<int>(23));
    l.Add(new Capsule<int>(42));

    // Have it sorted
    List<Capsule<int>> l2 = GetSortedReturnList(l);

    // Show result
    foreach (Capsule<int> ci in l2)
        Console.WriteLine(ci.Item);

    Console.ReadLine();
}

A simple list of two ints that are encapsulated in a generic type. The generic type just makes sure that two items can be compared:

internal class Capsule<T> : IComparable<Capsule<T>>
{
    public T Item;

    public Capsule(T i)
    {
        Item = i;
    }

    public int CompareTo(Capsule<T> other)
    {
        return Item.ToString().CompareTo(other.Item.ToString());
    }
}

And this code causes the exception when being executed by the veiled assembly:

private static List<Capsule<T>> 

    GetSortedReturnList<T>(List<Capsule<T>> list)
{           
    int sortOrder = 1;
    list.Sort(
        delegate(Capsule<T> a, Capsule<T> b) 

          { return sortOrder * a.CompareTo(b); });

    return (list);
}

The sore point is the sortOrder variable being multiplied with the result of a.CompareTo(b) used in an anonymous method. Okay, you wouldn’t expect this code from a beginner, but it’s not rocket-science either. So we were very astonished and it took us quite some time to track this down and to believe our eyes. Maybe the problem with CodeVeil is related to the internal representation of anonymous method. When you use a constant instead of a variable

const int sortOrder = 1;

the problem disappears. I guess the compiler eliminates the constant and CodeVeil will not see it. But it remains, when you use a Linq-notation here:

list.Sort((a, b) => a.CompareTo(b)*sortOrder);

As said before we found this bug half a year ago. We contacted Xheo and received a very quick and friendly reply, but no solution. Seasons went by and several new versions of CodeVeil were released; but this bug is still present. At least in the stand-alone version.

But there is hope: The new beta preview of the DeployLX suite contains an obviously new version of CodeVeil and the bug is finally gone. And – by the way – this new version seems to offer a nice package of new feature and improved power! So, please, Mr. Xheo, release a new stand-alone-version of CodeVeil based on the new beta preview!