ast_sip_get_transport_states was returning a container of internal_state
objects instead of ast_sip_transport_state objects. This was causing
transport lookups to fail, most noticably in res_pjsip_nat, which
couldn't find the correct external addresses. This was causing contacts
to go out with internal ip addresses.
ASTERISK-25830 #close
Reported-by: Sean Bright
Change-Id: I1aee6a2fd46c42e8dd0af72498d17de459ac750e
The 'reload' mechanism actually involves closing the underlying
socket and calling the appropriate udp, tcp or tls start functions
again. Only outbound_registration, pubsub and session needed work
to reset the transport before sending requests to insure that the
pjsip transport didn't get pulled out from under them.
In my testing, no calls were dropped when a transport was changed
for any of the 3 transport types even if ip addresses or ports were
changed. To be on the safe side however, a new transport option was
added (allow_reload) which defaults to 'no'. Unless it's explicitly
set to 'yes' for a transport, changes to that transport will be ignored
on a reload of res_pjsip. This should preserve the current behavior.
Change-Id: I5e759850e25958117d4c02f62ceb7244d7ec9edf
It is possible when processing a SIP REGISTER request to have two
threads end up creating contact_status structures in sorcery.
contact_status is created using a "find or create" function. If two
threads call into this at the same time, each thread will fail to find
an existing contact_status, and so both will end up creating a new
contact status.
During testing, we would see sporadic failures because the
PJSIP_CONTACT() dialplan function would operate on a different
contact_status than what had been updated by res_pjsip/pjsip_options.
The fix here is two-fold:
1) The "find or create" function for contact_status now has a lock
around the entire operation. This way, if two threads attempt the
operation simultaneously, the first to get there will create the object,
and the second will find the object created by the first thread.
2) res_sorcery_memory has had its create callback updated so that it
will not allow for objects with duplicate IDs to be created.
Change-Id: I55b1460ff1eb0af0a3697b82d7c2bac9f6af5b97
load_module was just too hairy with every step having to clean up all
previous steps on failure.
Some of the pjproject init calls have now been moved to a separate
load_pjsip function and the unload_pjsip function was enhanced to clean
up everything if an error happened at any stage of the load process.
In the process, a bunch of missing pj_shutdowns, serializer_pool_shutdowns
and ast_threadpool_shutdowns were also corrected.
Change-Id: I5eec711b437c35b56605ed99537ebbb30463b302
Attempting to load a transport from realtime was forcing asterisk into an
infinite recursion loop. The first thing transport_apply did was to do a
sorcery retrieve by id for an existing transport of the same name. For files,
this just returns the previous object from res_sorcery_config's internal
container, if any. For realtime, the res_sourcery_realtime driver looks in the
database and finds the existing row but now it has to rehydrate it into a
sorcery object which means calling... transport_apply. And so it goes.
The main issue with loading from realtime (apart from the loop) was that
transport stores structures and pointers directly in the ast_sip_transport
structure instead of the separate ast_transport_state structure. This patch
separates those items into the ast_sip_transport_state structure. The pattern
is roughly the same as res_pjsip_outbound_registration.
Although all current usages of ast_sip_transport and ast_sip_transport_state
were modified to use the new ast_sip_get_transport_state API, the original
items are left in ast_sip_transport and kept updated to maintain ABI
compatability for third-party modules. They are marked as deprecated and
noted that they're now in ast_sip_transport_state.
ASTERISK-25606 #close
Reported-by: Martin Moučka
Change-Id: Ic7a836ea8e786e8def51fe3f8cce855ea54f5f19
Added new global option (regcontext) to pjsip. When set, Asterisk will
dynamically create and destroy a NoOp priority 1 extension
for a given endpoint who registers or unregisters with us.
ASTERISK-25670 #close
Reported-by: Daniel Journo
Change-Id: Ib1530c5b45340625805c057f8ff1fb240a43ea62
On a system with multiple ip addresses in the same subnet, if a
transport is bound to a specific ip address and endpoint/media_address
is set, the SIP/SDP will have the correct address in all fields but
the rtp stream MAY still originate from one of the other ip addresses,
most probably the "primary" ip address. This happens because
res_pjsip_sdp_rtp/create_rtp always calls ast_instance_new with
the "all" ip address (0.0.0.0 or ::).
The new option causes res_pjsip_sdp_rtp/create_rtp to call
ast_rtp_instance_new with the endpoint's media_address (if specified)
instead of the "all" address. This causes the packets to originate from
the specified address.
ASTERISK-25632
ASTERISK-25637
Reported-by: Olivier Krief
Reported-by: Dan Journo
Change-Id: I3dfaa079e54ba7fb7c4fd1f5f7bd9509bbf8bd88
This reverts commit 0a9941de9d.
Matt,
This patch causes another problem and should not have been needed.
Before this patch, persistent_endpoint_contact_deleted_observer WAS
deleting the contact_status when ast_sip_location_delete_contact was
called. By deleting it yourself in ast_sip_location_delete_contact
it was gone before the observer could run and the observer therefore
was throwing an error and not sending stasis/AMI/statsd messages.
So, I don't think this was the cause of your original issue. I also
had verified the contact AMI and statsd lifecycle and it was working.
I'll double check now though.
ASTERISK-25675
Reported-by: Daniel Journo
Change-Id: Ib586a6b7f90acb641b0c410f659743ab90e84f1a
A deadlock was observed where the monitor thread was stuck, therefore
resulting in no incoming SIP traffic being processed.
The problem occurred when two 200 OK responses arrived in response to a
terminating NOTIFY request sent from Asterisk. The first 200 OK was
dispatched to a threadpool worker, who locked the corresponding
transaction. The second 200 OK arrived, resulting in the monitor thread
locking the dialog. At this point, the two threads are at odds, because
the monitor thread attempts to lock the transaction, and the threadpool
thread loops attempting to try to lock the dialog.
In this case, the fix is to not have the monitor thread attempt to hold
both the dialog and transaction locks at the same time. Instead, we
release the dialog lock before attempting to lock the transaction.
There have also been some debug messages added to the process in an
attempt to make it more clear what is going on in the process.
ASTERISK-25668 #close
Reported by Mark Michelson
Change-Id: I4db0705f1403737b4360e33a8e6276805d086d4a
In 450579e908, a change was made that removed the deletion of the
'contact_status' object when a 'contact' object is deleted in sorcery.
This unfortunately means that the 'contact_status' object persists, even when
something has explicitly removed a contact. The result is that the state of
the contact will not be regenerated if that contact is re-created, and the
stale state will be reported/used for that contact. It also results in
no ContactStatusChanged events being generated for either ARI or AMI.
This patch restores the deletion logic that was removed. Doing so now
results in the expected events being generated again.
Change-Id: I28789a112e845072308b5b34522690e3faf58f07
pjproject < 2.5.0 will segfault on a tls transport if async_operations
is greater than 1. A runtime version check has been added to throw
an error if the version is < 2.5.0 and async_operations > 1.
To assist in the check, a new api "ast_compare_versions" was added
to utils which compares 2 major.minor.patch.extra version strings.
ASTERISK-25615 #close
Change-Id: I8e88bb49cbcfbca88d9de705496d6f6a8c938a98
Reported-by: George Joseph
Tested-by: George Joseph
Both transport and endpoint now check for the existence and readability
of tls certificate and key files before passing them on to pjproject.
This will cause the object to not load rather than waiting for pjproject
to discover that there's a problem when a session is attempted.
NOTE: chan_sip also uses ast_rtp_dtls_cfg_parse but it's located
in build_peer which is gigantic and I didn't want to disturb it.
Error messages will emit but it won't interrupt chan_sip loading.
ASTERISK-25618 #close
Change-Id: Ie43f2c1d653ac1fda6a6f6faecb7c2ebadaf47c9
Reported-by: George Joseph
Tested-by: George Joseph
See ASTERISK-25615.
If the transport protocol is tls and async_operations > 1, pjproject
will segfault if more than one operation is attempted on the same socket.
Until this is fixed upstream, a check has been added to throw an error
if a tls transport config has async_operations set to > 1.
ASTERISK-25615
Change-Id: I76b9a5b2a5a0054fe71ca5851e635f2dca7685a6
Reported-by: George Joseph
Tested-by: George Joseph
It will never be perfect or even pretty, mostly because of the differences
between static and dynamic contacts.
Created:
Can't use the contact or contact_status alloc functions
because the objects come and go regardless of the actual state.
Can't use the contact_apply_handler, ast_sip_location_add_contact or
a sorcery created handler because they only get called for dynamic
contacts. Similarly, permanent_uri_handler only gets called for
static contacts.
So, Matt had it right. :) ast_res_pjsip_find_or_create_contact_status is
the only place it can go and not have duplicated code. Both
permanent_uri_handler and contact_apply_handler call find_or_create.
Removed:
Can't use the destructors for the same reason as above. The only
place to put this is in persistent_endpoint_contact_deleted_observer
which I believe is the "correct" place but even that will handle only
dynamic contacts. This doesn't called on shutdown however. There is
no hook to use for static contacts that may be removed because of a
config change while asterisk is in operation.
I moved the cleanup of contact_status from ast_sip_location_delete_contact
to the handler as well.
Status Change and RTT:
Although they worked fine where they were (in update_contact_status) I
moved them to persistent_endpoint_contact_status_observer to make it
more consistent with removed. There was logic there already to detect
a state change.
Finally, fixed a nit in permanent_uri_handler rmudgett reported
eralier.
ASTERISK-25608 #close
Change-Id: I4b56e7dfc3be3baaaf6f1eac5b2068a0b79e357d
Reported-by: George Joseph
Tested-by: George Joseph
When 90d9a70789 was merged, it mostly tested dynamic contacts created as
a result of registering a PJSIP endpoint. Contacts generated in this
fashion typically have a long alphanumeric string as their object identifier,
which maps reasonably well for StatsD. Unfortunately, this doesn't work in the
general case. StatsD treats both '.' and ':' characters as special characters.
In particular, having a ':' appear in the middle of a StatsD metric will
result in the metric being rejected.
This causes some obvious issues with SIP URIs.
The StatsD API should not be responsible for escaping the metric name passed
to it. The metric is treated as a single long string, and it would be
challenging to know what to escape in the string passed to the function.
Likewise, we don't want to escape the metric in PJSIP, as that involves
overhead that is wasted when either res_statsd isn't loaded or enabled.
This patch takes an alternative approach. The Contact ID has been changed
to be "aor@@uri_hash" instead of "aor@@uri". This (a) won't contain any of the
aforementioned special characters, (b) can be done on Contact creation,
which has minimal impact on run-time performance, and (c) also conforms to an
earlier commit that changed the ID for dynamic contacts.
The downside of this is that StatsD users will have to map SHA1 hashes back to
the Contacts that are emitting the statistics. To that end, the CLI commands
have been updated to include the first 10 characters of the MD5 hash, which
should be enough to match what is shown in Graphite (or some other StatsD
backend).
ASTERISK-25595 #close
Change-Id: Ic674a3307280365b4a45864a3571c295b48a01e2
Reported-by: Matt Jordan
Tested-by: George Joseph
An earlier commit changed the id of dynamic contacts to contain
a hash instead of the uri. This patch updates status change
logging to show the aor/uri instead of the id. This required
adding the aor id to contact and contact_status and adding
uri to contact_status. The aor id gets added to contact and
contact_status in their allocators and the uri gets added to
contact_status in pjsip_options when the contact_status is
created or updated.
ASTERISK-25598 #close
Reported-by: George Joseph
Tested-by: George Joseph
Change-Id: I56cbec1d2ddbe8461367dd8b6da8a6f47f6fe511
This patch adds the ability to send StatsD statistics related to the
state of PJSIP contacts. This includes:
* A GUAGE statistic measuring the count of contacts in a particular state.
This measures how many contacts are reachable, unreachable, etc.
* The RTT time for each contact, if those contacts are qualified. This
provides StatsD engines useful time-based data about each contact.
ASTERISK-25571
Change-Id: Ib8378d73afedfc622be0643b87c542557e0b332c
In practical tests, we have seen certain taskprocessors, specifically
Stasis subscription taskprocessors, cross the recently-added high-water
mark and emit a warning. This high-water mark warning is only intended
to be emitted when things have tanked on the system and things are
heading south quickly. In the practical tests, the Stasis taskprocessors
sometimes had a max depth of 180 tasks in them, and Asterisk wasn't in
any danger at all.
As such, this ups the high-water mark to 500 tasks instead. It also
redefines the SIP threadpool request denial number to be a multiple of
the taskprocessor high-water mark.
Change-Id: Ic8d3e9497452fecd768ac427bb6f58aa616eebce
When the SIP threadpool is backed up with tasks, we send 503 responses
to ensure that we don't try to overload ourselves. The problem is that
we were not insuring that we were not trying to send a 503 to an
incoming SIP response.
This change makes it so that we only send the 503 on incoming requests.
Change-Id: Ie2b418d89c0e453cc6c2b5c7d543651c981e1404
We have observed situations where the SIP threadpool may become
deadlocked. However, because incoming traffic is still arriving, the SIP
threadpool's queue can continue to grow, eventually running the system
out of memory.
This change makes it so that incoming traffic gets rejected with a 503
response if the queue is backed up too much.
Change-Id: I4e736d48a2ba79fd1f8056c0dcd330e38e6a3816
When an endpoint is deleted (such as through an API), the persistent endpoint
currently continues to lurk around. While this isn't harmful from a memory
consumption perspective - as all persistent endpoints are reclaimed on
shutdown - it does cause Stasis endpoint related operations to continue
to believe that the endpoint may or may not exist.
This patch causes the persistent endpoint related to a PJSIP endpoint to be
destroyed if the PJSIP endpoint is deleted.
Change-Id: I85ac707b4d5e6aad882ac275b0c2e2154affa5bb
The contact_status Sorcery objects are currently not destroyed when a contact
is deleted. This causes the contact's last known RTT/status to be 'sticky'
when the contact itself may no longer exist. This patch causes the
contact_status objects associated with both dynamic and static contacts to
be destroyed if the AoR holding those contacts is also destroyed (or via
other paths where a contact may be deleted.)
Change-Id: I7feec8b9278cac3c5263a4c0483f4a0f3b62426e
During a stress test of subscriptions, a huge blast of
subscription-related traffic resulted in the threadpool expanding to a
ridiculous number of threads. The balooning of threads resulted in an
increase of memory, which led to a crash due to being out of memory.
An easy fix for the particular test was to limit the size of the
threadpool, thus reining in the amount of memory that would be used. It
was decided that there really is no downside to having a non-infinite
default value for the maximum size of the threadpool, so this change
introduces 50 threads as the maximum threadpool size for the SIP
threadpool.
ASTERISK-25513 #close
Reported by John Bigelow
Change-Id: If0b9514f1d9b172540ce1a6e2f2ffa1f2b6119be
When an AoR is created or destroyed dynamically, the scheduled OPTIONS
requests that qualify the contacts on the AoR are not necessarily started
or destroyed, particularly for persistent contacts created for that AoR.
This patch adds create/update/delete sorcery observers for an AoR, which
schedule/unschedule the qualifies as expected.
Change-Id: Ic287ed2e2952a7808ee068776fe966f9554bdf7d
Add the ability to filter output from pjsip list and show commands
using the "like" predicate like chan_sip.
For endpoints, aors, auths, registrations, identifyies and transports,
the modification was a simple change of an ast_sorcery_retrieve_by_fields
call to ast_sorcery_retrieve_by_regex. For channels and contacts a
little more work had to be done because neither of those objects are
true sorcery objects. That was just removing the non-matching object
from the final container. Of course, a little extra plumbing in the
common pjsip_cli code was needed to parse the "like" and pass the regex
to the get_container callbacks.
Some of the get_container code in res_pjsip_endpoint_identifier was also
refactored for simplicity.
ASTERISK-25477 #close
Reported by: Bryant Zimmerman
Tested by: George Joseph
Change-Id: I646d9326b778aac26bb3e2bcd7fa1346d24434f1
In a realtime based system with a limited number of threadpool threads
it is possible for a deadlock to occur. This happens when permanent
endpoint state is updated, which will cause database queries to be done.
These queries may result in URI validation being done which is done
synchronously using a PJSIP thread. If all PJSIP threads are in use
processing traffic they themselves may be blocked waiting to get the
permanent endpoint container lock when identifying an endpoint.
This change moves URI validation to occur at use time instead of
configuration time. While this comes at a cost of not seeing a problem
until you use it it does solve the underlying deadlock problem.
ASTERISK-25486 #close
Change-Id: I2d7d167af987d23b3e8199e4a68f3359eba4c76a
The default_from_user retrieval function was pulling the
default_from_user from the global configuration struct in an unsafe way.
If using a database as a backend configuration store, the global
configuration struct is short-lived, so grabbing a pointer from it
results in referencing freed memory.
The fix here is to copy the default_from_user value out of the global
configuration struct.
Thanks go to John Hardin for discovering this problem and proposing the
patch on which this fix is based.
ASTERISK-25390 #close
Reported by Mark Michelson
Change-Id: I6b96067a495c1259da768f4012d44e03e7c6148c
In the wild it is possible for Contact URIs to be quite long as
parameters can exist on them. This can present a problem when storing
them in the AstDB as the URI is used as part of the object name and
there is a fixed length limit for the AstDB. This will cause
the contact to not get stored.
This change uses the MD5 hash of the Contact URI as part of the
object name instead. This has a fixed length which is guaranteed
to not exceed the AstDB length limit.
ASTERISK-25295 #close
Change-Id: Ie8252a75331ca00b41b9f308f42cc1fbdf701a02
When an AoR is deleted by an external mechanism, such as through ARI, we
currently do not remove dynamic contacts that were created for that AoR as a
result of a received REGISTER request. As a result, re-creating the AoR will
cause the dynamic contact to be interpreted as a persistent contact, leading
to some rather strange state being created for the contacts/endpoints.
This patch adds a sorcery observer for the 'aor' object. When a delete is
issued on the underlying sorcery object, the observer is called, and all
contacts created and persisted in sorcery for that AoR are also removed. Note
that we don't want to perform this action when an AO2 object that is an AoR is
destroyed, as the AoR can still exist in the backing storage (and we would
thus be removing valid contacts from an AoR that still "exists".)
ASTERISK-25381 #close
Change-Id: I6697e51ef6b2858b5d63401f35dc378bb0f90328
When Asterisk sends an outbound SIP request, if there is no direct
reason to place a specific value for the username in the From header,
Asterisk would generate a UUID. For example, this would happen when
sending outbound OPTIONS requests when qualifying or when sending
outbound INVITE requests when originating (if no explicit caller ID were
provided). The issue is that some SIP providers reject these sorts of
requests with a "Name too long" error response.
This patch aims to fix this by changing the default outbound username in
From headers to "asterisk". This value can be overridden by changing the
default_from_user option in the global options if desired.
ASTERISK-25377 #close
Reported by Mark Michelson
Change-Id: I6a4d34a56ff73ff4f661b0075aeba5461b7f3190
When an endpoint is backed by a non-static conf file backend (such as
the AstDB or Realtime), the 'auth' object may be returned as being an
empty string. Currently, res_pjsip will interpret that as being a valid
auth object, and will attempt to authenticate inbound requests. This
isn't desired; is an auth value is empty (which the name of an auth
object cannot be), we should instead interpret that as being an invalid
auth object and skip it.
ASTERISK-25339 #close
Change-Id: Ic32b0c6eb5575107d5164a8c40099e687cd722c7
The ast_sip_sanitize_xml function is used to sanitize
a string for placement into XML. This is done by examining
an input string and then appending values to an output
buffer. The function used by its implementation, strncat,
has specific behavior that was not taken into account.
If the size of the input string exceeded the available
output buffer size it was possible for the sanitization
function to write past the output buffer itself causing
a crash. The crash would either occur because it was
writing into memory it shouldn't be or because the resulting
string was not NULL terminated.
This change keeps count of how much remaining space is
available in the output buffer for text and only allows
strncat to use that amount.
Since this was exposed by the res_pjsip_pidf_digium_body_supplement
module attempting to send a large message the maximum allowed
message size has also been increased in it.
A unit test has also been added which confirms that the
ast_sip_sanitize_xml function is providing NULL terminated
output even when the input length exceeds the output
buffer size.
ASTERISK-25304 #close
Change-Id: I743dd9809d3e13d722df1b0509dfe34621398302
This change adds support for the 'rtp_timeout' and 'rtp_timeout_hold'
endpoint options. These allow the channel to be hung up if RTP
is not received from the remote endpoint for a specified number of
seconds.
ASTERISK-25259 #close
Change-Id: I3f39daaa7da2596b5022737b77799d16204175b9
This adds an "rtp_keepalive" option for PJSIP endpoints. Similar to the
chan_sip option, this specifies an interval, in seconds, at which we
will send RTP comfort noise frames. This can be useful for keeping RTP
sessions alive as well as keeping NAT associations alive during lulls.
ASTERISK-25242 #close
Reported by Mark Michelson
Change-Id: I3b9903d99e35fe5d0b53ecc46df82c750776bc8d
This patch fixes some bad default value handling in the following
settings:
* The 'message_context' and 'accountcode' settings are not mandatory. As
such, we can allow their stringfield values to be empty.
* The 'media_encryption' setting applies a default value of 'none' to
the setting, which it then can't parse or understand. Since the value
is documented to be 'no', this will now apply that as the default
value.
Change-Id: Ib9be7f97a7a5b9bc7aee868edf5acf38774cff83
The res_pjsip_mwi previously required a reload to set up the proper
subscriptions to allow unsolicited MWI to work. This change
makes it so the act of registering will also cause this to occur.
This is particularly useful if realtime is involved as no reload
needs to occur within Asterisk to cause the MWI information
to get sent.
ASTERISK-25180 #close
Change-Id: Id847b47de4b8b3ab8858455ccc2f07b0f915f252
Some phones send g.726 audio packed for AAL2, which differs from what is
recommended by RFC 3351. If Asterisk receives audio formatted as such when
negotiating g.726 then it sounds a bit distorted. Added an option to
res_pjsip_endpoint that allows g.726 negotiated audio to be treated as g.726
AAL2 packed.
ASTERISK-25158 #close
Reported by: Steve Pitts
Change-Id: Ie7e21f75493d7fe53e75e12c971e72f5afa33615
All send/receive processing for a SIP transaction needs to be done under
the same threadpool serializer to prevent reentrancy problems inside
pjproject when using an external DNS resolver to process messages for the
transaction.
* Add threadpool API call to get the current serializer associated with
the worker thread.
* Pick a serializer from a pool of default serializers if the caller of
res_pjsip.c:ast_sip_push_task() does not provide one.
This is a simple way to ensure that all outgoing SIP request messages are
processed under a serializer. Otherwise, any place where a pushed task is
done that would result in an outgoing out-of-dialog request would need to
be modified to supply a serializer. Serializers from the default
serializer pool are picked in a round robin sequence for simplicity.
A side effect is that the default serializer pool will limit the growth of
the thread pool from random tasks. This is not necessarily a bad thing.
* Made pjsip_resolver.c use the requesting thread's serializer to execute
the async callback.
* Made pjsip_distributor.c save the thread's serializer name on the
outgoing request tdata struct so the response can be processed under the
same serializer.
ASTERISK-25115 #close
Reported by: John Bigelow
Change-Id: Iea71c16ce1132017b5791635e198b8c27973f40a
* Fix query_set destruction before we are done kicking the queries off.
* Fixed no queries requested handling.
* Add empty queries request unit test.
* Added missing allocation check in ast_dns_query_set_add().
* Made initial pjsip resolving query vector slightly larger.
ASTERISK-25115
Reported by: John Bigelow
Change-Id: Ie8be8347d0992e93946d72b6e7b1299727b038f2
* Add some type casting so tv_usec can really be a long, instead of
some strange platform specific type.
* Add some .dylib style files to .gitignore.
* Switch from using -Xlinker to -Wl,. For [reasons unknown][], newer
versions of GCC, when compiling the Homebrew formula for Asterisk,
are not properly passing the -Xlinker options to the linker. Given
that -Wl, does exactly the [same thing][], and does it properly, this
patch changes the -Xlinker options to use -Wl, instead.
[reasons unknown]: http://bit.ly/1SUbEYx
[same thing]: https://gcc.gnu.org/onlinedocs/gcc/Link-Options.html
Change-Id: Id5e6b3c6cc86282ea5fca630dc3991137c5bf4dd
contact_apply_handler calls ast_res_pjsip_find_or_create_contact_status
to force the creation of a contact_status object whenever a new
contact is added but it didn't unref the returned object.
Added an ao2_cleanup(status) to plug the leak.
ASTERISK-25141
Change-Id: Icc1401cae142855a1abc86ab5179dfb3ee861c40
Reported-by: Corey Farrell
The loop to find the first available contact of an endpoint grabbed
contact from the iterator, then checked for offline state. This
caused the first contact after the state was found to leak a reference.
ASTERISK-25141
Change-Id: Id0f1d87410fc63742db0594eb4b18b36e99aec08
George Joseph
Mark Michelson
Richard Mudgett
Daniel Journo
Matt Jordan
David M. Lee
Joshua Colp
Kevin Harwell
Corey Farrell