Posts Tagged ‘Terminal Server

11
Sep
08

Summer Time~~? (Day Light Saving)

My Outlook calendar shows different time!!

Oh,  its that time of the year eh?
I dont know about you, but it has been a heck of a ride for AUS based system admins/engineers for past few years. I must say, I HATE politicians, especially when they keep making DLS related changes in States, Australia wide, etc etc, we engineers must be running around patching “updated” version of the DLS patch.  You miss one then you get a call like one in above with threat to management or how crap we’re doing. “sigh”

Fortunately, I dont know about others, but I cant imagine people maintain “desktops” gee that would be a hassle for sure. (I do server only that means less number of patch maintenance than desktops)

If you havent updated or need to update your summer time patch on MS windows based server, I think you should check here

Now I need to go check 100+ servers before summer time kicks in. <sarcasm>YAY!</sarcasm>

Advertisements
10
Sep
08

System hung? or dead?

You gotta be kidding me, another dead server?

Well past 4 wks or more, several servers were locking up in the middle of the day or morning (lucky not at the night as I am on-call for some days), especially when there are several users on the server. Sadly, by the time issue were reported by help desk to me, i have no remote access to the server.
Using ilo doesn’t work as I do not see login screen also remote procedures (such as event logs) just doesn’t responds. (You know what do to when you’re in this stage.. boot the server and get complaints from users+managers, in other word SHIT happens)

Complete LOCK up will be much better or even BSOD will give me less grief. BUT no, as RPC never times out nor IMASRV.EXE never dies but somehow responds by itself, Data collector redirects users to dead server even worse, user gets disconnected but can not be redirect to other “working” servers.

Yeah, 2nd blow to me. (again, I curse at the air and apologies to users) until the problem server is booted, or IMA finally realise (after 1-5mins) server is truly DEAD, user is STUCK at middle of no where.

FIrst thing first, what do you do? several options (not reboot)
1 read the event log (yes general rule of thumb but not in this case, event log was long dead by the time issue is discovered)
2 gather perfdata (well, using RM, I have started to gather more metrics such as memory, thread, etc etc)
again, by the time issue is reported, perfmon too is dead and data seems all green on the RM graph, yay.. no perfmon..)
3 login! (went to data centre to see if I can log on physically, but no.. Login screen doesnt come up)
4 telnet ? (screw security, I have enabled telnet service on the all the server and tried to login, but no it failed too)
5 RDP, ICA, VNC, Radmin, Ilo (list continues) no REMOTE tool works, regardless.

By the time I’ve reach step 5, more than 3 wks has passes and I have around 10 reports(incidents) and equally same number of issues were reported but never logged (call me stupid but never tracked them as I thought problem can be one off)

6, login to ALL the servers and wait for problem to hit.
Yes this worked and I have FINALLY observed server dying in front of me eyes.
Task Manager just stop responding, Process Explorer was hung when I flip the screen to problem server, explorer was frozen too, no new process could be started. BUT funny enough GUI was not frozen and I actually didnt get kicked out(disconnected)

Problem seems to be under LOW level somewhere I got no idea to.

Also, I had force system crash tool ready to kick in order to gather crash dump.

http://support.citrix.com/article/ctx111072

Well, you may have guessed by now, this tool didnt work either. It failed to crash and simply complain that files were not found.

7 last minute jump… call MS.

I didnt know what MS was going to say, but they gave me a light of how to crash server using NMI.
By then I was jumping in joy (ppl in office were looking at me like some weirdo.)

Finally! I can CRASH the system!!!

well rest if history, I’ve obtained debug analysis from MS that registry lock may be the cause and apply the patch from MS KB 935926

How annoying….




September 2017
M T W T F S S
« Jan    
 123
45678910
11121314151617
18192021222324
252627282930  

Greyeye Tweets