Maintenance Guidelines
~~~~~~~~~~~~~~~~~~~~~~
The CCNC will test the Node circuit from the 202T at the Host site to the 202T at the Node site. Since Host to Node (CCNC to MMOC) circuits are official company services, the CCNC will refer all Node circuit troubles to the SSC/MAC. The SSC/MAC is responsible for the testing and follow up to restoration of these circuit troubles.
Although Node to PSAP circuit are official services, the MMOC will refer PSAP circuit troubles to the appropriate SSC/MAC. The SSC/MAC is responsible for testing and follow up to restoration of PSAP circuit troubles.
The SSC/MAC will also receive reports from CRSAB/IMC(s) on subscriber 911 troubles when they are not line troubles. The SSC/MAC is responsible for testing and restoration of these troubles.
Maintenance responsibilities are as follows:
B R U C E S T E R L I N G — T H E H A C K E R C R A C K D O W N
NOT FOR COMMERCIAL USE
2 6 6
SCC* Voice Network (ANI to PSAP)
SCC responsible for tandem switch
SSIM/I&M PSAP Equipment (Modems, CIU's, sets) Vendor PSAP Equipment (when CPE)
SSC/MAC PSAP to Node circuits, and tandem to PSAP voice circuits (EMNT)
MMOC Node site (Modems, cables, etc)
Note: All above work groups are required to resolve troubles by interfacing with appropriate work groups for resolution.
The Switching Control Center (SCC) is responsible for E911/1AESS translations in tandem central offices. These translations route E911 calls, selective transfer, default routing, speed calling, etc., for each PSAP. The SCC is also responsible for troubleshooting on the voice network (call originating to end office tandem equipment).
For example, ANI failures in the originating offices would be a responsibility of the SCC.
Recent Change Memory Administration Center (RCMAC) performs the daily tandem translation updates (recent change) for routing of individual telephone numbers.
Recent changes are generated from service order activity (new service, address changes, etc.) and compiled into a daily file by the E911 Center (ALI/DMS E911 Computer).
SSIM/I&M is responsible for the installation and repair of PSAP equipment.
PSAP equipment includes ANI Controller, ALI Controller, data sets, cables, sets, and other peripheral equipment that is not vendor owned. SSIM/I&M is responsible for establishing maintenance test kits, complete with spare parts for PSAP maintenance. This includes test gear, data sets, and ANI/ALI Controller parts.
Special Services Center (SSC) or Major Account Center (MAC) serves as the trouble reporting contact for all (PSAP) troubles reported by customer. The SSC/MAC refers troubles to proper organizations for handling and tracks status of troubles, escalating when necessary. The SSC/MAC will close out troubles with customer. The SSC/MAC will analyze all troubles and tracks
"chronic" PSAP troubles.
Corporate Communications Network Center (CCNC) will test and refer troubles on all node to host circuits. All E911 circuits are classified as official company property.
The Minicomputer Maintenance Operations Center (MMOC) maintains the E911
(ALI/DMS) computer hardware at the Host site. This MMOC is also responsible for monitoring the system and reporting certain PSAP and system problems to the local MMOC's, SCC's or SSC/MAC's. The MMOC personnel also operate software programs that maintain the TN data base under the direction of the E911 Center. The maintenance of the NODE computer (the interface B R U C E S T E R L I N G — T H E H A C K E R C R A C K D O W N
NOT FOR COMMERCIAL USE
2 6 7
between the PSAP and the ALI/DMS computer) is a function of the MMOC at the NODE site. The MMOC's at the NODE sites may also be involved in the testing of NODE to Host circuits. The MMOC will also assist on Host to PSAP
and data network related troubles not resolved through standard trouble clearing procedures.
Installation And Maintenance Center (IMC) is responsible for referral of E911
subscriber troubles that are not subscriber line problems.
E911 Center - Performs the role of System Administration and is responsible for overall operation of the E911 computer software. The E911 Center does A-Z trouble analysis and provides statistical information on the performance of the system.
This analysis includes processing PSAP inquiries (trouble reports) and referral of network troubles. The E911 Center also performs daily processing of tandem recent change and provides information to the RCMAC for tandem input. The E911 Center is responsible for daily processing of the ALI/DMS
computer data base and provides error files, etc. to the Customer Services department for investigation and correction. The E911 Center participates in all system implementations and ongoing maintenance effort and assists in the development of procedures, training and education of information to all groups.
Any group receiving a 911 trouble from the SSC/MAC should close out the trouble with the SSC/MAC or provide a status if the trouble has been referred to another group. This will allow the SSC/MAC to provide a status back to the customer or escalate as appropriate.
Any group receiving a trouble from the Host site (MMOC or CCNC) should close the trouble back to that group.
The MMOC should notify the appropriate SSC/MAC when the Host, Node, or all Node circuits are down so that the SSC/MAC can reply to customer reports that may be called in by the PSAPs. This will eliminate duplicate reporting of troubles. On complete outages the MMOC will follow escalation procedures for a Node after two (2) hours and for a PSAP after four (4) hours. Additionally the MMOC will notify the appropriate SSC/MAC when the Host, Node, or all Node circuits are down.
The PSAP will call the SSC/MAC to report E911 troubles. The person reporting the E911 trouble may not have a circuit I.D. and will therefore report the PSAP name and address. Many PSAP troubles are not circuit specific. In those instances where the caller cannot provide a circuit I.D., the SSC/MAC
will be required to determine the circuit I.D. using the PSAP profile. Under no circumstances will the SSC/MAC Center refuse to take the trouble. The E911
trouble should be handled as quickly as possible, with the SSC/MAC providing as much assistance as possible while taking the trouble report from the caller.
The SSC/MAC will screen/test the trouble to determine the appropriate handoff organization based on the following criteria: B R U C E S T E R L I N G — T H E H A C K E R C R A C K D O W N
NOT FOR COMMERCIAL USE
2 6 8
PSAP equipment problem: SSIM/I&M
Circuit problem: SSC/MAC
Voice network problem: SCC (report trunk group number) Problem affecting multiple PSAPs (No ALI report from all PSAPs): Contact the MMOC to check for NODE or Host computer problems before further testing.
The SSC/MAC will track the status of reported troubles and escalate as appropriate. The SSC/MAC will close out customer/company reports with the initiating contact. Groups with specific maintenance responsibilities, defined above, will investigate "chronic" troubles upon request from the SSC/MAC and the ongoing maintenance subcommittee.
All "out of service" E911 troubles are priority one type reports. One link down to a PSAP is considered a priority one trouble and should be handled as if the PSAP was isolated.
The PSAP will report troubles with the ANI controller, ALI controller or set equipment to the SSC/MAC.
NO ANI: Where the PSAP reports NO ANI (digital display screen is blank) ask if this condition exists on all screens and on all calls. It is important to differentiate between blank screens and screens displaying 911-00XX, or all zeroes.
When the PSAP reports all screens on all calls, ask if there is any voice contact with callers. If there is no voice contact the trouble should be referred to the SCC immediately since 911 calls are not getting through which may require alternate routing of calls to another PSAP.
When the PSAP reports this condition on all screens but not all calls and has voice contact with callers, the report should be referred to SSIM/I&M for dispatch. The SSC/MAC should verify with the SCC that ANI is pulsing before dispatching SSIM.
When the PSAP reports this condition on one screen for all calls (others work fine) the trouble should be referred to SSIM/I&M for dispatch, because the trouble is isolated to one piece of equipment at the customer premise.
An ANI failure (i.e. all zeroes) indicates that the ANI has not been received by the PSAP from the tandem office or was lost by the PSAP ANI controller.
The PSAP may receive "02" alarms which can be caused by the ANI controller logging more than three all zero failures on the same trunk. The PSAP
has been instructed to report this condition to the SSC/MAC since it could indicate an equipment trouble at the PSAP which might be affecting all subscribers calling into the PSAP. When all zeroes are being received on all calls or "02" alarms continue, a tester should analyze the condition to determine the appropriate action to be taken. The tester must perform cooperative testing with the SCC when there appears to be a problem on the Tandem-PSAP
trunks before requesting dispatch.
B R U C E S T E R L I N G — T H E H A C K E R C R A C K D O W N
NOT FOR COMMERCIAL USE
2 6 9
When an occasional all zero condition is reported, the SSC/MAC should dispatch SSIM/I&M to routine equipment on a "chronic" troublesweep.
The PSAPs are instructed to report incidental ANI failures to the BOC on a PSAP inquiry trouble ticket (paper) that is sent to the Customer Services E911 group and forwarded to E911 center when required. This usually involves only a particular telephone number and is not a condition that would require a report to the SSC/MAC. Multiple ANI failures which our from the same end office (XX denotes end office), indicate a hard trouble condition may exist in the end office or end office tandem trunks. The PSAP will report this type of condition to the SSC/MAC and the SSC/MAC should refer the report to the SCC responsible for the tandem office. NOTE: XX is the ESCO
(Emergency Service Number) associated with the incoming 911 trunks into the tandem. It is important that the C/MAC tell the SCC what is displayed at the PSAP (i.e. 911-0011) which indicates to the SCC which end office is in trouble.
Note: It is essential that the PSAP fill out inquiry form on every ANI failure.
The PSAP will report a trouble any time an address is not received on an address display (screen blank) E911 call. (If a record is not in the 911 data base or an ANI failure is encountered, the screen will provide a display noticing such condition). The SSC/MAC should verify with the PSAP whether the NO ALI condition is on one screen or all screens.
When the condition is on one screen (other screens receive ALI information) the SSC/MAC will request SSIM/I&M to dispatch.
If no screens are receiving ALI information, there is usually a circuit trouble between the PSAP and the Host computer. The SSC/MAC should test the trouble and refer for restoral.
Note: If the SSC/MAC receives calls from multiple PSAP's, all of which are receiving NO ALI, there is a problem with the Node or Node to Host circuits or the Host computer itself. Before referring the trouble the SSC/MAC should call the MMOC to inquire if the Node or Host is in trouble.
Alarm conditions on the ANI controller digital display at the PSAP are to be reported by the PSAP's. These alarms can indicate various trouble conditions so the SSC/MAC should ask the PSAP if any portion of the E911 system is not functioning properly.
The SSC/MAC should verify with the PSAP attendant that the equipment's primary function is answering E911 calls. If it is, the SSC/MAC should request a dispatch SSIM/I&M. If the equipment is not primarily used for E911, then the SSC/MAC should advise PSAP to contact their CPE vendor.
Note: These troubles can be quite confusing when the PSAP has vendor equipment mixed in with equipment that the BOC maintains. The Marketing representative should provide the SSC/MAC information concerning any unusual or exception items where the PSAP should contact their vendor. This information should be included in the PSAP profile sheets.
B R U C E S T E R L I N G — T H E H A C K E R C R A C K D O W N
NOT FOR COMMERCIAL USE
2 7 0
ANI or ALI controller down: When the host computer sees the PSAP equipment down and it does not come back up, the MMOC will report the trouble to the SSC/MAC; the equipment is down at the PSAP, a dispatch will be required.
PSAP link (circuit) down: The MMOC will provide the SSC/MAC with the circuit ID that the Host computer indicates in trouble. Although each PSAP has two circuits, when either circuit is down the condition must be treated as an emergency since failure of the second circuit will cause the PSAP to be isolated.
Any problems that the MMOC identifies from the Node location to the Host computer will be handled directly with the appropriate MMOC(s)/CCNC.
Note: The customer will call only when a problem is apparent to the PSAP.
When only one circuit is down to the PSAP, the customer may not be aware there is a trouble, even though there is one link down, notification should appear on the PSAP screen. Troubles called into the SSC/MAC from the MMOC or other company employee should not be closed out by calling the PSAP since it may result in the customer responding that they do not have a trouble. These reports can only be closed out by receiving information that the trouble was fixed and by checking with the company employee that reported the trouble. The MMOC personnel will be able to verify that the trouble has cleared by reviewing a printout from the host.
When the CRSAB receives a subscriber complaint (i.e., cannot dial 911) the RSA should obtain as much information as possible while the customer is on the line.
For example, what happened when the subscriber dialed 911? The report is automatically directed to the IMC for subscriber line testing. When no line trouble is found, the IMC will refer the trouble condition to the SSC/MAC.
The SSC/MAC will contact Customer Services E911 Group and verify that the subscriber should be able to call 911 and obtain the ESN. The SSC/MAC will verify the ESN via 2SCCS. When both verifications match, the SSC/MAC will refer the report to the SCC responsible for the 911 tandem office for investigation and resolution. The MAC is responsible for tracking the trouble and informing the IMC when it is resolved.
For more information, please refer to E911 Glossary of Terms.