Help! My ELB isn’t serving files!
Whoa! My back-end instances work but not the ELB!
Hey! I can’t get the ELB to work!
These are among the most common Elastic Load Balancer problems raised on the Amazon EC2 Discussion Forums. Inspired by Eric Hammond’s indispensible article Solving “I can’t connect to my server on Amazon EC2”, here is a helpful guide to debugging these common ELB issues, as well as a utility to perform sanity tests on your own ELBs.
Questions to Answer
You’re trying to figure out what’s wrong and you need to know where to start looking. Or, you’re posting your problem on the AWS forums and you want help as quickly as possible. The best way to help yourself or to get help quickly is to examine the basic facts of your situation. Here are some questions to answer for yourself and in your forum post:
- What is the output of
elb-describe-lbs elbName --show-xml? This gives the basic details of the ELB, which are critical to diagnosing any problem. If you are posting to the forums and want to keep the DNS name of the ELB private then obscure it in the output. One reason to obscure the DNS name is to prevent readers from accessing your ELB-based service. However, this precaution does not add any security because the DNS information is public, and – presumably – you are using a DNS CNAME entry to integrate the ELB into your domain’s DNS. - What is the output of
elb-describe-instance-health elbName? This provides crucial information about the health of the instances. - What resource are you trying to access via the ELB and what tool are you using to access it and from what location? The resource will likely be a URL of the form
http://ELB-DNS-Name/index.htmlor maybehttps://ELB-DNS-Name/index.html, or it might be “I’m running a POP server on port 1234”. The tool you’re using to access it is most likely a browser or HTTP client (Firefox, or wget), or possibly “Microsoft Outlook version 5.4”. The location is either “my local machine” or “an EC2 instance”. Also, can you access the same resource when you connect directly to a back-end instance via its public IP address or host name from a client outside EC2? A public-facing URL pointing directly to a back-end instance looks like this:http://ec2-123-213-123-31.compute-1.amazonaws.com/index.html. And, can you access the same resource when you connect directly to a back-end instance via its private IP address or host name from another instance within EC2? Such a URL looks like this:http://domU-12-31-34-00-69-B9.compute-1.internal/index.html. - Can you access the health check resource directly via the ELB DNS name, and via the back-end instance’s public IP address, and via the back-end instance’s private IP address? If your health check is configured with
target=HTTP:8080/check.htmlthen try to accesshttp://ELB-DNS-Name:8080/check.html(which is via the ELB) andhttp://ec2-123-213-123-31.compute-1.amazonaws.com:8080/check.html(which is via the instance’s public IP address) andhttp://domU-12-31-34-00-69-B9.compute-1.internal:8080/check.html(which is via the instance’s private IP address, and only accessible from within EC2). - What are the security groups and availability zones for each instance in the ELB? This is visible in the output of
ec2-describe-instances i-11111111 i-22222222 ...As above, you might want to obscure the public and private DNS names of these instances in the output. - Can all the back-end instances receive traffic on the instance ports of the ELB listeners and the health check? This can be checked from the output of
ec2-describe-group groupName1 groupName2 ...for all the groups shown in question 5’sec2-describe-instancescommand. - Do logs on your back-end instances show any connections from ELB?
Okay, now that you know what information is important to diagnosing the problem, here is a look at some of the common gotchas, how to detect them, and how to fix them. These descriptions refer to the above questions by number.Common problems and solutions include:
- Security groups on back-end instances don’t allow access to the instance ports and health check port. Back-end instances must have all ports on which they receive traffic from the ELB (#1) open to CIDR 0.0.0.0/0 in one of their associated security groups (#6). Fix this by changing the permissions on the security groups associated with the instances. Note: this fix takes effect within a few seconds and does not require launching new instance or rebooting existing instances.
- Back-end instances are not healthy (
InService). When an instance fails the health check (#1) it is marked asOutOfService(#2) and the ELB does not route traffic to it anymore. To fix this you need to determine why the ELB cannot access the health check resource. Note: there is currently a bug in ELB where instances initially are marked asInServicewhen added to the ELB, until they fail the health check. So you’ll want to make sure you’ve given ELB enough time to detect a failed health check. Update August 2010: AWS folks say that bug has been fixed. - An availability zone is enabled on the ELB but has no healthy back-end instances. If you have an availability zone enabled for your ELB (#1) but no healthy instances in that availability zone (#5 and #2), you’ll get 503 Gateway Timeout or other errors. Fix this by adding an instance in that availability zone to the ELB or disabling that availability zone for the ELB.
- You cannot see a requested resource (#3) or the health check URL (#4) using the ELB DNS name. In this case, check that the URL exists on the back-end instances and look at the back-end instance’s logs (#7) to see if the ELB forwarded your connection or not. If you can see the requested resource using the public address of a back-end instance then check the instance’s security groups (#6) to see that they grant access to the instance’s port.
- The health check port is not the same as listener target port (#1). While this does not necessarily indicate a problem, for most ELBs the health check should use the same port as one of the listeners. Setting up your ELB to have a health check performed on a different port than the load-balanced traffic is perfectly valid, but you likely want the health check to use the same path that the load-balanced traffic takes to reach your app (and also to exercise a representative set of features used by your app).
I will update this article with new common issues as they appear.
An ELB Sanity Test Utility
If you have your thinking cap on you’ll notice that detecting the first three of the common ELB problems can be automated. Here is an ELB sanity test utility for linux which automates these tests. Save it or download it as follows:
curl -o elb-sanity-test.tar.gz -L https://github.com/shlomoswidler/elb-sanity-test/releases/download/v0.1/elb-sanity-test.tar.gz
Next, unpack it:
tar xzf elb-sanity-test.tar.gz
cd elb-sanity-test
Next, set up the utility with your credentials. Edit the elb-sanity-test script file, setting AWS_CREDENTIAL_FILE to point to a file containing your AWS credentials in the following format:
AWSAccessKeyId=your-Access-Key-Id
AWSSecretKey=your-Secret-Key
The above is the same format that can be used to specify your AWS credentials for the ELB API Tools (see the README.TXT and credential-file-path.template file in the ELB API Tools bundle).
To run the ELB sanity test:
cd elb-sanity-test
./elb-sanity-test
Here is sample output showing an ELB that passes the sanity test:
$ ./elb-sanity-test JUnit version 4.5 Test: all instances have their Security Groups defined to allow access to the ELB listener port Load Balancer: someLB ELB someLB has a listener that uses instance-port 8080 and instance i-360ef05e has that TCP port open to the world. ELB someLB has a listener that uses instance-port 8081 and instance i-360ef05e has that TCP port open to the world. Test: all ELBs have a HealthCheck on a port that the listener directs traffic to Load Balancer: someLB ELB someLB has a configured HealthCheck on listener port 8080 Test: all ELBs have InService instances in each configured availability zone Load Balancer: somLB ELB someLB has InService instances in each configured availability zone Time: 5.22 Tests run: 3, Failures: 0
The elb-sanity-test utility performs the following sanity tests on every ELB defined in your account:
- All instances have their security groups defined to allow access to the ELB listener port.
- All ELBs have a health check on a port that the listener directs traffic to.
- All ELBs have healthy instances in each configured availability zone.
If a sanity test fails the utility shows a very verbose error message explaining what is wrong.
Some notes about the elb-sanity-test bundle:
- The utility is written in Java, which is also required for the ELB tools. If you can run the ELB API Tools, you already have all the prerequisites to run this sanity test.
- The bundle includes source code and is licensed under the Apache License, Version 2.0.
- The bundle includes all dependency jars necessary to run the script. It uses the JUnit framework and the Typica library.
I would be happy to re-bundle the utility to include a .bat or .cmd file to make it easy to run the script on Windows. If you write one, please add it it in the comments and I’ll include it.
Getting Further Help
If you still have an ELB issue after trying the above advice and the elb-sanity-test utility, please post in the AWS EC2 forum. Questions about the elb-sanity-test utility specifically or about this article are welcome in the comments below.
Update 15 September 2009: Ylastic integrated my elb-sanity-test script into their EC2 management dashboard.
Update 11 October 2009: elb-sanity-test has been released as part of the open-source ec2-elb-tests project hosted on Google Code. And, if you use this utility, please subscribe to the ec2-elb-tests Google Group.
Update 21 July 2014: The project has moved to be hosted on GitHub.
38 replies on “Solving Common ELB Problems with a Sanity Test”
Excellent post! Very useful tips on what to look for while debugging ELBs. We just integrated this approach into our management UI – http://vimeo.com/6592989
thanks!
Very useful post! I just saw that Ylastic integrated your sanity check approach into their dashboard. Nice!
I wanted to share a problem I had recently. Because of some current ELB issues, it is possible to cause an ELB to get into an inconsistent state by terminating one of its instances without first deregistering it from the ELB. See the thread Mysterious "server=0" content returned by ELB on the AWS forums.
Also, if you are trying to read client IP addresses on a server behind an ELB, see my blog post: Amazon ELB – Capturing Client IP Address
Is it possible to edit a load balancer to forward additional ports? For example we launched our aws ELB forwarding port 80, and now we want to add https support and forward 443 as well. Do I need to create a new ELB or can I edit my existing one?
@Logan Green,
There's no API to modify the listeners of an ELB. So you'll need to create a new ELB with both HTTP:80 and TCP:443 listeners specified at creation time.
I am getting the NoClassDefFoundError error. can you help please?
./elb-sanity-test
JUnit version 4.5
.E
Time: 0.003
There was 1 failure:
1) initializationError(shlomo.ec2.elb.ELBSanityTest)
java.lang.NoClassDefFoundError: com/xerox/amazonws/ec2/EC2Exception
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getMethod0(Class.java:2670)
at java.lang.Class.getMethod(Class.java:1603)
at org.junit.internal.builders.SuiteMethodBuilder.hasSuiteMethod(SuiteMethodBuilder.java:20)
at org.junit.internal.builders.SuiteMethodBuilder.runnerForClass(SuiteMethodBuilder.java:13)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:29)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:57)
at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:93)
at org.junit.runners.model.RunnerBuilder.runners(RunnerBuilder.java:84)
at org.junit.runners.Suite.(Suite.java:66)
at org.junit.runner.Request.classes(Request.java:68)
at org.junit.runner.JUnitCore.run(JUnitCore.java:107)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:88)
at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:54)
at org.junit.runner.JUnitCore.main(JUnitCore.java:46)
Caused by: java.lang.ClassNotFoundException: com.xerox.amazonws.ec2.EC2Exception
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
… 17 more
FAILURES!!!
Tests run: 1, Failures: 1
@Raj,
Thanks for reporting this. I’ve created Issue 2 for this issue, and I fixed the problem and uploaded a new jar.
Please try again.
bacon status: saved
thanks for these protips, i had an AZ assigned with no instances in it, was getting otherwise unexplainable 503s.
cheers
@Andrew Bullock,
Glad these techniques helped you.
If you are in a VPC,
Make sure that if you create a new subnet to hold your ELB, that you add it to the VPC routing table(s). Otherwise traffic from the outside will not be able to get to the ELB.
@ty,
Thanks for the important addition!
[…] Here is a great post on troubleshooting your AWS ELB. […]
I’m a beginner at ELB. One hurdle that took me a while . . . In Route 53 I set up a CNAME that pointed to the ELB A record. This did not work! My wget using the CNAME resulted in “503 Service Unavailable: Back-end server is at capacity”.
Everything was resolved when I used an ALIAS in Route53 per AWS recommendations instead of a CNAME.
RFM. 🙂
@Karla,
Thanks for the tip. ELB has matured significantly now that it can be integrated with Route 53 – and so has the documentation 😉
Shlomo,
Does the sanity tool honor the AWS_ELB_URL env variable? I have it set to https://elasticloadbalancing.eu-west-1.amazonaws.com, and I still get information about my US-East instances.
Thanks.
@jbells,
elb-sanity-test was written at a time when ELB was only available in us-east-1.
The code is hosted on Google Code – which I have grown to dislike, and I’d move it to Github if that makes it easier for you to submit a patch adding support for additional regions.
As an aside: If I were to write this again from scratch today, I’d probably write it in Ruby and use the new awscli tool, and keep it in Github.
Hey I’m getting below error. Please help me to fix this issue.
There was 1 failure:
1) testHealthCheckIsListenerInstancePort(shlomo.ec2.elb.ELBSanityTest)
java.lang.AssertionError: ELB devicecheck has a TCP HealthCheck on port 80 corresponding to a listener of the same protocol and port
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at shlomo.ec2.elb.ELBSanityTest.testHealthCheckIsListenerInstancePort(ELBSanityTest.java:205)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:73)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:46)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runners.Suite.runChild(Suite.java:115)
at org.junit.runners.Suite.runChild(Suite.java:23)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:116)
at org.junit.runner.JUnitCore.run(JUnitCore.java:107)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:88)
at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:54)
at org.junit.runner.JUnitCore.main(JUnitCore.java:46)
FAILURES!!!
Tests run: 3, Failures: 1
Many Thanks,
Madhu Neal
@Madhu Neal,
This error indicates that your ELB is configured to direct TCP traffic to port 80 of the instances, but the ELB’s health check does not check TCP connections to port 80. Normally you want the health check to use the same port and protocol as the ELB traffic. Does your ELB’s health check use TCP on port 80?
Shlomo,
Thanks for providing sanity test tool.
Can you help please how to develop sanity test tool to check SQS message in perl/php?
@Hareesh,
You’re welcome!
What kind of SQS sanity testing are you looking for?
I am getting the following error, what does it mean?
There was 1 failure:
1) shlomo.ec2.elb.ELBSanityTest
com.xerox.amazonws.ec2.LoadBalancingException: Client error : (RequestExpired) Request has expired. Timestamp date: 2014-07-18T16:02:00Z
at com.xerox.amazonws.ec2.LoadBalancing.makeRequestInt(LoadBalancing.java:389)
at com.xerox.amazonws.ec2.LoadBalancing.describeLoadBalancers(LoadBalancing.java:257)
at com.xerox.amazonws.ec2.LoadBalancing.describeLoadBalancers(LoadBalancing.java:237)
at shlomo.ec2.elb.ELBSanityTest.setUpBeforeClass(ELBSanityTest.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runners.Suite.runChild(Suite.java:115)
at org.junit.runners.Suite.runChild(Suite.java:23)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:116)
at org.junit.runner.JUnitCore.run(JUnitCore.java:107)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:88)
at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:54)
at org.junit.runner.JUnitCore.main(JUnitCore.java:46)
FAILURES!!!
Tests run: 0, Failures: 1
Thanks
@Avin Agrawal,
Looks like the request to AWS’s API to list ELBs times out. Usually this happens when your machine cannot communicate with AWS’s API. Do any AWS command-line tools work on your machine?
Try this one: http://aws.amazon.com/cli/
Thank you for your advice
No the command line tools don’t work and I always get this error
Signature expired: is now earlier than
What could be the reason for this?
@Avin Agrawal,
That error usually means that AWS’s ELB API thinks the time is significantly different than the time on the computer you’re calling it from.
The solution is to fix the time on the machine running the code. Look at this thread for some tips:
https://forums.aws.amazon.com/message.jspa?messageID=529837#529837
That worked. Thanks a lot for your help. Now, I am successfully able to run the tests.
Hi Shlomo,
Your advice regarding the elastic load balancer was very useful. Do you by any chance have experience with node js. I have been able to setup nodejs on amazon linux ec2 instances but I am looking for a way to keep the node server running even after the instance reboots. I have come across many methods to do that, including the crontab, forever and rc.local, but all of them work only for ubuntu. Do you know any method by which this can be achieved for amazon linux. I will be very thankful if you could help me with this. Thank you.
@Avin Agrawal,
Happy to hear that. You’re welcome.
Amazon Linux is a RedHat-derivative, like CentOS, so the technique presented at the end of this article should work: https://www.exratione.com/2011/07/running-a-nodejs-server-as-a-service-using-forever/
That helped. Thanks
Hi, great tool. Just one question…
I’ve been getting a PeerNotVerified exception:
1) shlomo.ec2.elb.ELBSanityTest
com.xerox.amazonws.ec2.LoadBalancingException: peer not verified
at com.xerox.amazonws.ec2.LoadBalancing.makeRequestInt(LoadBalancing.java:395)
at com.xerox.amazonws.ec2.LoadBalancing.describeLoadBalancers(LoadBalancing.java:257)
at com.xerox.amazonws.ec2.LoadBalancing.describeLoadBalancers(LoadBalancing.java:237)
at shlomo.ec2.elb.ELBSanityTest.setUpBeforeClass(ELBSanityTest.java:83)
at java.lang.reflect.Method.invoke(libgcj.so.10)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runners.Suite.runChild(Suite.java:115)
at org.junit.runners.Suite.runChild(Suite.java:23)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:116)
at org.junit.runner.JUnitCore.run(JUnitCore.java:107)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:88)
at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:54)
at org.junit.runner.JUnitCore.main(JUnitCore.java:46)
Did some googling and I can’t find any information on this error message. For what it’s worth, the ELB does actually seem to be working fine.
Either way, thanks for writing this little tool. I’m sure it will come in very handy down the line!
@Dave Thomas,
That exception looks like what happens when the SSL certificate used by the server cannot be verified. It would be strange for ELB’s servers to be using an unverifiable SSL cert.
Do you get this error when using the new Ruby-based version of the code?
Thanks for the fast reply!
I actually don’t have SSL set up on the ELB (or the EC2 that it’s talking to) yet. That might be what is causing it then?
@Dave Thomas,
It’s not a message about the SSL cert on your ELB listener. It’s about the SSL cert on the ELB API endpoint, which the code is calling to request the ELB’s configuration details. It’s an error that AWS is responsible for, not you.
Hi,
I have created one ELB in VPC in public subnets and add 2 instance of each subnet.
my port configuration is 80 -> 2443 and 9443 -> 9443
In security group of ELB i have open port 80 for all.
i am performing health check on port 2443 as http:2443/healthcheck/
I allowed security group of ELB on port 2443 and 9443 on each instance.
but my instances are showing out of service.
i have removed it and again attched it but it is also not working
please help me make instance in inservice state.
@rahul goyal,
What does the sanity test report when you run it?
Thanks for the great tool! I’m getting the error:
Instance i-xxxxx does not have these ports open to listen to ELB traffic: 80.
However, my load balancer has two listeners: 80 -> 80 and 443 -> 80. Also, my back-end instance security group has port 80 open to my load balancer. I’m not sure why I’m getting this error, because to me it seems port 80 is open everywhere it ought to be. Any ideas?
@Nick,
Can you run the tool with the
--debugoption and share the output?I eventually figured out the issue…Django has an ALLOWED_HOSTS setting that serves as a whitelist of hosts, and the ELB uses a changing internal IP when performing the health check (which was not originally in my ALLOWED_HOSTS). Thus, the request would not go through.
Thanks for the excellent post.
While I try to perform a sanity check for the elb’s in our environment. I’m getting following exception. Can you help us in fixing the issue. Thanks in Advance.
./elb-sanity-test
JUnit version 4.5
E
Time: 0.285
There was 1 failure:
1) shlomo.ec2.elb.ELBSanityTest
com.xerox.amazonws.ec2.LoadBalancingException: peer not authenticated
at com.xerox.amazonws.ec2.LoadBalancing.makeRequestInt(LoadBalancing.java:395)
at com.xerox.amazonws.ec2.LoadBalancing.describeLoadBalancers(LoadBalancing.java:257)
at com.xerox.amazonws.ec2.LoadBalancing.describeLoadBalancers(LoadBalancing.java:237)
at shlomo.ec2.elb.ELBSanityTest.setUpBeforeClass(ELBSanityTest.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runners.Suite.runChild(Suite.java:115)
at org.junit.runners.Suite.runChild(Suite.java:23)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at org.junit.runner.JUnitCore.run(JUnitCore.java:116)
at org.junit.runner.JUnitCore.run(JUnitCore.java:107)
at org.junit.runner.JUnitCore.runMain(JUnitCore.java:88)
at org.junit.runner.JUnitCore.runMainAndExit(JUnitCore.java:54)
at org.junit.runner.JUnitCore.main(JUnitCore.java:46)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at sun.security.ssl.SSLSessionImpl.getPeerCertificates(SSLSessionImpl.java:431)
at org.apache.http.conn.ssl.AbstractVerifier.verify(AbstractVerifier.java:128)
at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:339)
at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:123)
at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:147)
at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:108)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554)
at com.xerox.amazonws.common.AWSQueryConnection.makeRequest(AWSQueryConnection.java:470)
at com.xerox.amazonws.ec2.LoadBalancing.makeRequestInt(LoadBalancing.java:387)
… 27 more
FAILURES!!!
Tests run: 0, Failures: 1
See my response above to @Dave Thomas.
Also, please try the master branch of the code on GitHub, which has been rewritten in ruby and is easier to use.