AR# 3785: V1.4 COREGEN: Blank DOS box appears when COREGen is invoked with Windows Display set to "True Color", entire program hangs; Windows NT.
AR# 3785
V1.4 COREGEN: Blank DOS box appears when COREGen is invoked with Windows Display set to "True Color", entire program hangs; Windows NT.
Description
Keywords: coregen, hang, display, DOS, True Color, Windows NT
Urgency: standard
General Description: COREGen may be unable to start up on some machines if the Windows Display setting is set to "True Color", and Resolution is set to higher than 800x600. A blank DOS box pops up to the foreground instead of staying resident in the Taskbar, and the system just hangs.
The problem has been observed on a PC with a 21" Viewsonic monitor controlled by an ATI Graphics Pro card with 4MB of video memory. The main system contained 64MB of RAM, and was running Windows NT 4.0 Service Pack 3.
Solution
1
The fix is to change the colors setting for the DISPLAY to 256 colors. After changing this setting, COREGen should then able to proceed forward and display the GUI window, and otherwise run normally.
2
The problem appears related to the following scenario documented on the Microsoft Web site, and may be similar to a problem with Windows NT's interaction with Internet Explorer and other programs that use the Java routine, GetDIBits.
Microsoft's suggested fix is to wait for the next, post- version 3 Service Pack that contains this fix.
The location of the documentation on this problem can be found at the Microsoft web site at
DOCUMENT:Q168748 [winnt] TITLE :Java Applets Cause IE 3.02 to Stop Responding w/ SP3 PRODUCT :Microsoft Windows NT PROD/VER:4.00 OPER/SYS:WINDOWS KEYWORDS:kbbug4.00 kbfile kbfix4.00
------------------------------------------------------------------------- The information in this article applies to:
- Microsoft Windows NT Workstation version 4.0 - Microsoft Windows NT Server version 4.0 - Microsoft Internet Explorer version 3.02 -------------------------------------------------------------------------
SYMPTOMS ========
Internet Explorer version 3.02 may hang when connecting to a Web link that contains a Java application after you have installed Windows NT version 4.0 Service Pack 3. The application will stop responding only if the Display Properties Color Palette is set to True Color.
CAUSE =====
In Windows NT version 4.0 Service Pack 3 the GetDIBits API does not fill the bit fields when a BI_RGB (uncompressed) DIB is returned on a True Color (32 bpp) display. This GDI change was made because the bit fields are not required in this case. As a side effect, this change may cause a regression in applications that expect a valid RGB mask on a True Color display.
The Java virtual machine included with Internet Explorer 3.02 calls GetDIBits when initializing the display palette. The Java virtual machine expects a valid, signed RGB mask value larger than 0x80h and, if a value less than 0x80h is received, it doubles the mask value in a tight loop until it is larger than 0x80h. Because Service Pack 3 GetDIBits do not fill the bit fields when a BI_RGB is returned, applications that evaluate the RGB mask receive an unlikely result that can be negative (for example, 0xffffffe0h.) When the Java virtual machine evaluates the RGB mask and receives a negative value, it loops forever and causes Internet Explorer to stop responding.
It is important to note that the Java virtual machine should be treating the RGB Mask as an unsigned integer. There are some conditions where an ATI video card returns a valid RGB Mask value of 0xff000000h causing the Java virtual machine to loop forever because it is not treating the mask as an unsigned integer. A separate fix for the Java virtual machine issue with ATI video cards in True Color will be available in Internet Explorer 4.0. The Java code is licensed from Sun Microsystems, Inc.
RESOLUTION ==========
To work around this problem, do not use the True Color palette in Display Properties. To resolve this problem, obtain the hotfix mentioned below or wait for the next service pack.
Win32k.sys was modified to return a valid RGB color mask on True Color display systems.
Java-fix has been superseded by Getadmin-fix. You can find the latest hotfix at the following Microsoft ftp site:
WARNING: If you install the original (archived) version of this hotfix AFTER you apply the later version, your system may become unusable. Microsoft does not recommend you install the original hotfix after applying the later version.
STATUS ======
Microsoft has confirmed this to be a problem in Windows NT version 4.0. A supported fix is now available, but is not fully regression-tested and should be applied only to systems experiencing this specific problem. Unless you are severely impacted by this specific problem, Microsoft recommends that you wait for the next Service Pack that contains this fix. Contact Microsoft Product Support Services for more information.