2008-12-13 08:36:35 +00:00
|
|
|
/*
|
|
|
|
* Asterisk -- An open source telephony toolkit.
|
|
|
|
*
|
|
|
|
* Copyright (C) 2008, Digium, Inc.
|
|
|
|
*
|
|
|
|
* Tilghman Lesher <tlesher@digium.com>
|
|
|
|
*
|
|
|
|
* See http://www.asterisk.org for more information about
|
|
|
|
* the Asterisk project. Please do not directly contact
|
|
|
|
* any of the maintainers of this project for assistance;
|
|
|
|
* the project provides a web site, mailing lists and IRC
|
|
|
|
* channels for your use.
|
|
|
|
*
|
|
|
|
* This program is free software, distributed under the terms of
|
|
|
|
* the GNU General Public License Version 2. See the LICENSE file
|
|
|
|
* at the top of the source tree.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*! \file
|
|
|
|
*
|
|
|
|
* \brief String manipulation API
|
|
|
|
*
|
|
|
|
* \author Tilghman Lesher <tilghman@digium.com>
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*** MAKEOPTS
|
2010-02-16 00:52:45 +00:00
|
|
|
<category name="MENUSELECT_CFLAGS" displayname="Compiler Flags" positive_output="yes">
|
|
|
|
<member name="DEBUG_OPAQUE" displayname="Change ast_str internals to detect improper usage" touch_on_change="include/asterisk/strings.h">
|
2008-12-13 08:36:35 +00:00
|
|
|
<defaultenabled>yes</defaultenabled>
|
|
|
|
</member>
|
|
|
|
</category>
|
|
|
|
***/
|
|
|
|
|
2012-06-15 16:20:16 +00:00
|
|
|
/*** MODULEINFO
|
|
|
|
<support_level>core</support_level>
|
|
|
|
***/
|
|
|
|
|
2008-12-13 08:36:35 +00:00
|
|
|
#include "asterisk.h"
|
|
|
|
|
sorcery/res_pjsip: Refactor for realtime performance
There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.
A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0. One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.
This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare. The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.
They do now.
The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator. For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'". If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.
The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container. However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.
So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function. Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex. If the operator is like or regex, the
right string should be a %-pattern or a regex expression. If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.
To use this new function on ast_variables, 2 new functions were added to
config.c. One that compares 2 ast_variables, and one that compares 2
ast_variable lists. The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list. The latter will traverse the right list and return true if all
the variables in it match the left list.
Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines. The realtime backend just passes
the variable list unaltered to the engine. The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.
Only one more change to sorcery was done... A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)
Now on to res_pjsip...
pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors. Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.
res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.
res_pjsip_registrar_expire was completely refactored. It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them. A new
contact_expiration_check_interval was added to global with a default of
30 seconds.
Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.
There are still objects that can't be filtered at the database like
identifies, transports, and registrations. These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.
Back to allow_unqualified_fetch. If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :) Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache. Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts. It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.
Example sorcery.conf:
[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error
ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer
Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67
2016-03-08 21:55:30 +00:00
|
|
|
#include <regex.h>
|
2008-12-13 08:36:35 +00:00
|
|
|
#include "asterisk/strings.h"
|
|
|
|
#include "asterisk/pbx.h"
|
|
|
|
|
|
|
|
/*!
|
|
|
|
* core handler for dynamic strings.
|
|
|
|
* This is not meant to be called directly, but rather through the
|
|
|
|
* various wrapper macros
|
|
|
|
* ast_str_set(...)
|
|
|
|
* ast_str_append(...)
|
|
|
|
* ast_str_set_va(...)
|
|
|
|
* ast_str_append_va(...)
|
|
|
|
*/
|
|
|
|
|
2016-08-19 23:19:28 +00:00
|
|
|
#ifdef __AST_DEBUG_MALLOC
|
2010-02-17 21:22:40 +00:00
|
|
|
int __ast_debug_str_helper(struct ast_str **buf, ssize_t max_len,
|
2009-03-11 00:29:59 +00:00
|
|
|
int append, const char *fmt, va_list ap, const char *file, int lineno, const char *function)
|
|
|
|
#else
|
2010-02-17 21:22:40 +00:00
|
|
|
int __ast_str_helper(struct ast_str **buf, ssize_t max_len,
|
2008-12-13 08:36:35 +00:00
|
|
|
int append, const char *fmt, va_list ap)
|
2009-03-11 00:29:59 +00:00
|
|
|
#endif
|
2008-12-13 08:36:35 +00:00
|
|
|
{
|
2015-10-19 20:28:46 +00:00
|
|
|
int res;
|
|
|
|
int added;
|
|
|
|
int need;
|
2008-12-18 15:25:15 +00:00
|
|
|
int offset = (append && (*buf)->__AST_STR_LEN) ? (*buf)->__AST_STR_USED : 0;
|
2008-12-13 08:36:35 +00:00
|
|
|
va_list aq;
|
|
|
|
|
2015-10-19 20:28:46 +00:00
|
|
|
if (max_len < 0) {
|
|
|
|
max_len = (*buf)->__AST_STR_LEN; /* don't exceed the allocated space */
|
|
|
|
}
|
|
|
|
|
2008-12-13 08:36:35 +00:00
|
|
|
do {
|
|
|
|
va_copy(aq, ap);
|
2008-12-18 15:25:15 +00:00
|
|
|
res = vsnprintf((*buf)->__AST_STR_STR + offset, (*buf)->__AST_STR_LEN - offset, fmt, aq);
|
2015-10-19 20:28:46 +00:00
|
|
|
va_end(aq);
|
|
|
|
|
|
|
|
if (res < 0) {
|
|
|
|
/*
|
|
|
|
* vsnprintf write to string failed.
|
|
|
|
* I don't think this is possible with a memory buffer.
|
|
|
|
*/
|
|
|
|
res = AST_DYNSTR_BUILD_FAILED;
|
|
|
|
added = 0;
|
|
|
|
break;
|
|
|
|
}
|
2008-12-13 08:36:35 +00:00
|
|
|
|
|
|
|
/*
|
2015-10-19 20:28:46 +00:00
|
|
|
* vsnprintf returns how much space we used or would need.
|
|
|
|
* Remember that vsnprintf does not count the nil terminator
|
|
|
|
* so we must add 1.
|
2008-12-13 08:36:35 +00:00
|
|
|
*/
|
2015-10-19 20:28:46 +00:00
|
|
|
added = res;
|
|
|
|
need = offset + added + 1;
|
|
|
|
if (need <= (*buf)->__AST_STR_LEN
|
|
|
|
|| (max_len && max_len <= (*buf)->__AST_STR_LEN)) {
|
|
|
|
/*
|
|
|
|
* There was enough room for the string or we are not
|
|
|
|
* allowed to try growing the string buffer.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Reallocate the buffer and try again. */
|
|
|
|
if (max_len == 0) {
|
|
|
|
/* unbounded, give more room for next time */
|
|
|
|
need += 16 + need / 4;
|
|
|
|
} else if (max_len < need) {
|
|
|
|
/* truncate as needed */
|
|
|
|
need = max_len;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (
|
2016-08-19 23:19:28 +00:00
|
|
|
#ifdef __AST_DEBUG_MALLOC
|
2015-10-19 20:28:46 +00:00
|
|
|
_ast_str_make_space(buf, need, file, lineno, function)
|
2009-03-11 00:29:59 +00:00
|
|
|
#else
|
2015-10-19 20:28:46 +00:00
|
|
|
ast_str_make_space(buf, need)
|
2009-03-11 00:29:59 +00:00
|
|
|
#endif
|
2015-10-19 20:28:46 +00:00
|
|
|
) {
|
|
|
|
ast_log_safe(LOG_VERBOSE, "failed to extend from %d to %d\n",
|
|
|
|
(int) (*buf)->__AST_STR_LEN, need);
|
|
|
|
|
|
|
|
res = AST_DYNSTR_BUILD_FAILED;
|
|
|
|
break;
|
2008-12-13 08:36:35 +00:00
|
|
|
}
|
|
|
|
} while (1);
|
2015-10-19 20:28:46 +00:00
|
|
|
|
|
|
|
/* Update space used, keep in mind truncation may be necessary. */
|
|
|
|
(*buf)->__AST_STR_USED = ((*buf)->__AST_STR_LEN <= offset + added)
|
|
|
|
? (*buf)->__AST_STR_LEN - 1
|
|
|
|
: offset + added;
|
|
|
|
|
|
|
|
/* Ensure that the string is terminated. */
|
|
|
|
(*buf)->__AST_STR_STR[(*buf)->__AST_STR_USED] = '\0';
|
2008-12-13 08:36:35 +00:00
|
|
|
|
|
|
|
return res;
|
|
|
|
}
|
|
|
|
|
2010-02-17 21:22:40 +00:00
|
|
|
char *__ast_str_helper2(struct ast_str **buf, ssize_t maxlen, const char *src, size_t maxsrc, int append, int escapecommas)
|
2008-12-13 08:36:35 +00:00
|
|
|
{
|
|
|
|
int dynamic = 0;
|
2008-12-18 15:25:15 +00:00
|
|
|
char *ptr = append ? &((*buf)->__AST_STR_STR[(*buf)->__AST_STR_USED]) : (*buf)->__AST_STR_STR;
|
2008-12-13 08:36:35 +00:00
|
|
|
|
2009-02-18 23:51:35 +00:00
|
|
|
if (maxlen < 1) {
|
|
|
|
if (maxlen == 0) {
|
|
|
|
dynamic = 1;
|
|
|
|
}
|
2008-12-18 15:25:15 +00:00
|
|
|
maxlen = (*buf)->__AST_STR_LEN;
|
2008-12-13 08:36:35 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
while (*src && maxsrc && maxlen && (!escapecommas || (maxlen - 1))) {
|
|
|
|
if (escapecommas && (*src == '\\' || *src == ',')) {
|
|
|
|
*ptr++ = '\\';
|
|
|
|
maxlen--;
|
2008-12-18 15:25:15 +00:00
|
|
|
(*buf)->__AST_STR_USED++;
|
2008-12-13 08:36:35 +00:00
|
|
|
}
|
|
|
|
*ptr++ = *src++;
|
|
|
|
maxsrc--;
|
|
|
|
maxlen--;
|
2008-12-18 15:25:15 +00:00
|
|
|
(*buf)->__AST_STR_USED++;
|
|
|
|
|
2009-04-02 15:14:22 +00:00
|
|
|
if ((ptr >= (*buf)->__AST_STR_STR + (*buf)->__AST_STR_LEN - 3) ||
|
|
|
|
(dynamic && (!maxlen || (escapecommas && !(maxlen - 1))))) {
|
2009-02-16 23:48:54 +00:00
|
|
|
char *oldbase = (*buf)->__AST_STR_STR;
|
2008-12-18 15:25:15 +00:00
|
|
|
size_t old = (*buf)->__AST_STR_LEN;
|
|
|
|
if (ast_str_make_space(buf, (*buf)->__AST_STR_LEN * 2)) {
|
2008-12-13 08:36:35 +00:00
|
|
|
/* If the buffer can't be extended, end it. */
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
/* What we extended the buffer by */
|
|
|
|
maxlen = old;
|
2009-02-16 23:48:54 +00:00
|
|
|
|
2009-02-18 23:51:35 +00:00
|
|
|
ptr += (*buf)->__AST_STR_STR - oldbase;
|
2008-12-13 08:36:35 +00:00
|
|
|
}
|
|
|
|
}
|
2009-04-01 20:13:28 +00:00
|
|
|
if (__builtin_expect(!maxlen, 0)) {
|
2008-12-13 08:36:35 +00:00
|
|
|
ptr--;
|
|
|
|
}
|
|
|
|
*ptr = '\0';
|
2008-12-18 15:25:15 +00:00
|
|
|
return (*buf)->__AST_STR_STR;
|
2008-12-13 08:36:35 +00:00
|
|
|
}
|
|
|
|
|
2013-08-02 14:08:34 +00:00
|
|
|
static int str_hash(const void *obj, const int flags)
|
|
|
|
{
|
|
|
|
return ast_str_hash(obj);
|
|
|
|
}
|
|
|
|
|
2017-01-19 15:05:36 +00:00
|
|
|
static int str_sort(const void *lhs, const void *rhs, int flags)
|
|
|
|
{
|
|
|
|
if ((flags & OBJ_SEARCH_MASK) == OBJ_SEARCH_PARTIAL_KEY) {
|
|
|
|
return strncmp(lhs, rhs, strlen(rhs));
|
|
|
|
} else {
|
|
|
|
return strcmp(lhs, rhs);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-08-02 14:08:34 +00:00
|
|
|
static int str_cmp(void *lhs, void *rhs, int flags)
|
|
|
|
{
|
2017-01-19 15:05:36 +00:00
|
|
|
int cmp = 0;
|
|
|
|
|
|
|
|
if ((flags & OBJ_SEARCH_MASK) == OBJ_SEARCH_PARTIAL_KEY) {
|
|
|
|
cmp = strncmp(lhs, rhs, strlen(rhs));
|
|
|
|
} else {
|
|
|
|
cmp = strcmp(lhs, rhs);
|
|
|
|
}
|
|
|
|
|
|
|
|
return cmp ? 0 : CMP_MATCH;
|
2013-08-02 14:08:34 +00:00
|
|
|
}
|
|
|
|
|
2015-03-30 02:39:57 +00:00
|
|
|
//struct ao2_container *ast_str_container_alloc_options(enum ao2_container_opts opts, int buckets)
|
|
|
|
struct ao2_container *ast_str_container_alloc_options(enum ao2_alloc_opts opts, int buckets)
|
2013-08-02 14:08:34 +00:00
|
|
|
{
|
2017-01-19 15:05:36 +00:00
|
|
|
return ao2_container_alloc_hash(opts, 0, buckets, str_hash, str_sort, str_cmp);
|
2013-08-02 14:08:34 +00:00
|
|
|
}
|
2013-05-21 18:00:22 +00:00
|
|
|
|
2013-08-02 14:08:34 +00:00
|
|
|
int ast_str_container_add(struct ao2_container *str_container, const char *add)
|
|
|
|
{
|
2014-01-14 21:46:50 +00:00
|
|
|
char *ao2_add;
|
2013-08-02 14:08:34 +00:00
|
|
|
|
2014-01-14 21:46:50 +00:00
|
|
|
/* The ao2_add object is immutable so it doesn't need a lock of its own. */
|
|
|
|
ao2_add = ao2_alloc_options(strlen(add) + 1, NULL, AO2_ALLOC_OPT_LOCK_NOLOCK);
|
2013-08-02 14:08:34 +00:00
|
|
|
if (!ao2_add) {
|
|
|
|
return -1;
|
|
|
|
}
|
2014-01-14 21:46:50 +00:00
|
|
|
strcpy(ao2_add, add);/* Safe */
|
2013-08-02 14:08:34 +00:00
|
|
|
|
|
|
|
ao2_link(str_container, ao2_add);
|
2014-01-14 21:46:50 +00:00
|
|
|
ao2_ref(ao2_add, -1);
|
2013-08-02 14:08:34 +00:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
void ast_str_container_remove(struct ao2_container *str_container, const char *remove)
|
|
|
|
{
|
2014-01-14 21:46:50 +00:00
|
|
|
ao2_find(str_container, remove, OBJ_SEARCH_KEY | OBJ_NODATA | OBJ_UNLINK);
|
2013-08-02 14:08:34 +00:00
|
|
|
}
|
2014-08-07 19:26:32 +00:00
|
|
|
|
|
|
|
char *ast_generate_random_string(char *buf, size_t size)
|
|
|
|
{
|
|
|
|
int i;
|
|
|
|
|
|
|
|
for (i = 0; i < size - 1; ++i) {
|
|
|
|
buf[i] = 'a' + (ast_random() % 26);
|
|
|
|
}
|
|
|
|
buf[i] = '\0';
|
|
|
|
|
|
|
|
return buf;
|
|
|
|
}
|
sorcery/res_pjsip: Refactor for realtime performance
There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.
A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0. One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.
This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare. The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.
They do now.
The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator. For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'". If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.
The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container. However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.
So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function. Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex. If the operator is like or regex, the
right string should be a %-pattern or a regex expression. If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.
To use this new function on ast_variables, 2 new functions were added to
config.c. One that compares 2 ast_variables, and one that compares 2
ast_variable lists. The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list. The latter will traverse the right list and return true if all
the variables in it match the left list.
Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines. The realtime backend just passes
the variable list unaltered to the engine. The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.
Only one more change to sorcery was done... A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)
Now on to res_pjsip...
pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors. Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.
res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.
res_pjsip_registrar_expire was completely refactored. It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them. A new
contact_expiration_check_interval was added to global with a default of
30 seconds.
Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.
There are still objects that can't be filtered at the database like
identifies, transports, and registrations. These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.
Back to allow_unqualified_fetch. If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :) Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache. Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts. It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.
Example sorcery.conf:
[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error
ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer
Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67
2016-03-08 21:55:30 +00:00
|
|
|
|
|
|
|
int ast_strings_match(const char *left, const char *op, const char *right)
|
|
|
|
{
|
|
|
|
char *internal_op = (char *)op;
|
|
|
|
char *internal_right = (char *)right;
|
Use doubles instead of floats for conversions when comparing strings.
In 13.9.0, there was an issue where PJSIP contacts added to an AOR would
be deleted at seemingly random times.
One reason this was happening was because of an operation to retrieve
the contacts whose expiration time was less than or equal to the current
time. When retrieving existing contacts, the contact's expiration time
and the current time were converted from a string to a float, and those
two floats were compared.
On some systems, including mine, this conversion was horribly off. For
instance, I could regularly see the string "1463079214" get converted
into 1463079168.000000. When switching from using a float to using a
double, the conversion was as expected.
Why was the conversion to float off? My best guess is that the
conversion to float was attempting to store the entire value in the 23
bit significand of the IEEE-754 floating point number. In particular, if
you take only the 23 most significant bits of 1463079214, you get the
messed up 1463079168 that we were seeing in the conversion. It likely
was possible to get a more precise value by composing the number using
an exponent, but the conversion did not work that way. With a double,
you have a 52 bit significand, allowing the entire value to fit there,
and thereby allowing an accurate conversion.
ASTERISK-26007 #close
Reported by Greg Siemon
Change-Id: I83ca7944aae8b7cd994b254c78ec02411d321070
2016-05-12 19:36:25 +00:00
|
|
|
double left_num;
|
|
|
|
double right_num;
|
sorcery/res_pjsip: Refactor for realtime performance
There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.
A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0. One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.
This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare. The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.
They do now.
The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator. For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'". If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.
The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container. However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.
So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function. Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex. If the operator is like or regex, the
right string should be a %-pattern or a regex expression. If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.
To use this new function on ast_variables, 2 new functions were added to
config.c. One that compares 2 ast_variables, and one that compares 2
ast_variable lists. The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list. The latter will traverse the right list and return true if all
the variables in it match the left list.
Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines. The realtime backend just passes
the variable list unaltered to the engine. The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.
Only one more change to sorcery was done... A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)
Now on to res_pjsip...
pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors. Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.
res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.
res_pjsip_registrar_expire was completely refactored. It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them. A new
contact_expiration_check_interval was added to global with a default of
30 seconds.
Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.
There are still objects that can't be filtered at the database like
identifies, transports, and registrations. These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.
Back to allow_unqualified_fetch. If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :) Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache. Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts. It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.
Example sorcery.conf:
[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error
ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer
Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67
2016-03-08 21:55:30 +00:00
|
|
|
int scan_numeric = 0;
|
|
|
|
|
|
|
|
if (!(left && right)) {
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (ast_strlen_zero(op)) {
|
|
|
|
if (ast_strlen_zero(left) && ast_strlen_zero(right)) {
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (strlen(right) >= 2 && right[0] == '/' && right[strlen(right) - 1] == '/') {
|
|
|
|
internal_op = "regex";
|
|
|
|
internal_right = ast_strdupa(right);
|
|
|
|
/* strip the leading and trailing '/' */
|
|
|
|
internal_right++;
|
|
|
|
internal_right[strlen(internal_right) - 1] = '\0';
|
|
|
|
goto regex;
|
|
|
|
} else {
|
|
|
|
internal_op = "=";
|
|
|
|
goto equals;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!strcasecmp(op, "like")) {
|
|
|
|
char *tok;
|
|
|
|
struct ast_str *buffer = ast_str_alloca(128);
|
|
|
|
|
|
|
|
if (!strchr(right, '%')) {
|
|
|
|
return !strcmp(left, right);
|
|
|
|
} else {
|
|
|
|
internal_op = "regex";
|
|
|
|
internal_right = ast_strdupa(right);
|
|
|
|
tok = strsep(&internal_right, "%");
|
|
|
|
ast_str_set(&buffer, 0, "^%s", tok);
|
|
|
|
|
|
|
|
while ((tok = strsep(&internal_right, "%"))) {
|
|
|
|
ast_str_append(&buffer, 0, ".*%s", tok);
|
|
|
|
}
|
|
|
|
ast_str_append(&buffer, 0, "%s", "$");
|
|
|
|
|
|
|
|
internal_right = ast_str_buffer(buffer);
|
|
|
|
/* fall through to regex */
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
regex:
|
|
|
|
if (!strcasecmp(internal_op, "regex")) {
|
|
|
|
regex_t expression;
|
|
|
|
int rc;
|
|
|
|
|
|
|
|
if (regcomp(&expression, internal_right, REG_EXTENDED | REG_NOSUB)) {
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
rc = regexec(&expression, left, 0, NULL, 0);
|
|
|
|
regfree(&expression);
|
|
|
|
return !rc;
|
|
|
|
}
|
|
|
|
|
|
|
|
equals:
|
2017-10-06 01:23:31 +00:00
|
|
|
scan_numeric = (sscanf(left, "%lf", &left_num) > 0 && sscanf(internal_right, "%lf", &right_num) > 0);
|
sorcery/res_pjsip: Refactor for realtime performance
There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.
A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0. One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.
This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare. The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.
They do now.
The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator. For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'". If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.
The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container. However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.
So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function. Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex. If the operator is like or regex, the
right string should be a %-pattern or a regex expression. If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.
To use this new function on ast_variables, 2 new functions were added to
config.c. One that compares 2 ast_variables, and one that compares 2
ast_variable lists. The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list. The latter will traverse the right list and return true if all
the variables in it match the left list.
Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines. The realtime backend just passes
the variable list unaltered to the engine. The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.
Only one more change to sorcery was done... A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)
Now on to res_pjsip...
pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors. Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.
res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.
res_pjsip_registrar_expire was completely refactored. It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them. A new
contact_expiration_check_interval was added to global with a default of
30 seconds.
Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.
There are still objects that can't be filtered at the database like
identifies, transports, and registrations. These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.
Back to allow_unqualified_fetch. If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :) Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache. Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts. It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.
Example sorcery.conf:
[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error
ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer
Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67
2016-03-08 21:55:30 +00:00
|
|
|
|
|
|
|
if (internal_op[0] == '=') {
|
|
|
|
if (ast_strlen_zero(left) && ast_strlen_zero(internal_right)) {
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (scan_numeric) {
|
|
|
|
return (left_num == right_num);
|
|
|
|
} else {
|
|
|
|
return (!strcmp(left, internal_right));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (internal_op[0] == '!' && internal_op[1] == '=') {
|
|
|
|
if (scan_numeric) {
|
|
|
|
return (left_num != right_num);
|
|
|
|
} else {
|
|
|
|
return !!strcmp(left, internal_right);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (internal_op[0] == '<') {
|
|
|
|
if (scan_numeric) {
|
|
|
|
if (internal_op[1] == '=') {
|
|
|
|
return (left_num <= right_num);
|
|
|
|
} else {
|
|
|
|
return (left_num < right_num);
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
if (internal_op[1] == '=') {
|
|
|
|
return strcmp(left, internal_right) <= 0;
|
|
|
|
} else {
|
|
|
|
return strcmp(left, internal_right) < 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (internal_op[0] == '>') {
|
|
|
|
if (scan_numeric) {
|
|
|
|
if (internal_op[1] == '=') {
|
|
|
|
return (left_num >= right_num);
|
|
|
|
} else {
|
|
|
|
return (left_num > right_num);
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
if (internal_op[1] == '=') {
|
|
|
|
return strcmp(left, internal_right) >= 0;
|
|
|
|
} else {
|
|
|
|
return strcmp(left, internal_right) > 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2017-09-11 10:46:35 +00:00
|
|
|
char *ast_read_line_from_buffer(char **buffer)
|
|
|
|
{
|
|
|
|
char *start = *buffer;
|
|
|
|
|
|
|
|
if (!buffer || !*buffer || *(*buffer) == '\0') {
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
|
|
|
while (*(*buffer) && *(*buffer) != '\n' ) {
|
|
|
|
(*buffer)++;
|
|
|
|
}
|
|
|
|
|
|
|
|
*(*buffer) = '\0';
|
|
|
|
if (*(*buffer - 1) == '\r') {
|
|
|
|
*(*buffer - 1) = '\0';
|
|
|
|
}
|
|
|
|
(*buffer)++;
|
sorcery/res_pjsip: Refactor for realtime performance
There were a number of places in the res_pjsip stack that were getting
all endpoints or all aors, and then filtering them locally.
A good example is pjsip_options which, on startup, retrieves all
endpoints, then the aors for those endpoints, then tests the aors to see
if the qualify_frequency is > 0. One issue was that it never did
anything with the endpoints other than retrieve the aors so we probably
could have skipped a step and just retrieved all aors. But nevermind.
This worked reasonably well with local config files but with a realtime
backend and thousands of objects, this was a nightmare. The issue
really boiled down to the fact that while realtime supports predicates
that are passed to the database engine, the non-realtime sorcery
backends didn't.
They do now.
The realtime engines have a scheme for doing simple comparisons. They
take in an ast_variable (or list) for matching, and the name of each
variable can contain an operator. For instance, a name of
"qualify_frequency >" and a value of "0" would create a SQL predicate
that looks like "where qualify_frequency > '0'". If there's no operator
after the name, the engines add an '=' so a simple name of
"qualify_frequency" and a value of "10" would return exact matches.
The non-realtime backends decide whether to include an object in a
result set by calling ast_sorcery_changeset_create on every object in
the internal container. However, ast_sorcery_changeset_create only does
exact string matches though so a name of "qualify_frequency >" and a
value of "0" returns nothing because the literal "qualify_frequency >"
doesn't match any name in the objset set.
So, the real task was to create a generic string matcher that can take a
left value, operator and a right value and perform the match. To that
end, strings.c has a new ast_strings_match(left, operator, right)
function. Left and right are the strings to operate on and the operator
can be a string containing any of the following: = (or NULL or ""), !=,
>, >=, <, <=, like or regex. If the operator is like or regex, the
right string should be a %-pattern or a regex expression. If both left
and right can be converted to float, then a numeric comparison is
performed, otherwise a string comparison is performed.
To use this new function on ast_variables, 2 new functions were added to
config.c. One that compares 2 ast_variables, and one that compares 2
ast_variable lists. The former is useful when you want to compare 2
ast_variables that happen to be in a list but don't want to traverse the
list. The latter will traverse the right list and return true if all
the variables in it match the left list.
Now, the backends' fields_cmp functions call ast_variable_lists_match
instead of ast_sorcery_changeset_create and they can now process the
same syntax as the realtime engines. The realtime backend just passes
the variable list unaltered to the engine. The only gotcha is that
there's no common realtime engine support for regex so that's been noted
in the api docs for ast_sorcery_retrieve_by_fields.
Only one more change to sorcery was done... A new config flag
"allow_unqualified_fetch" was added to reg_sorcery_realtime.
"no": ignore fetches if no predicate fields were supplied.
"error": same as no but emit an error. (good for testing)
"yes": allow (the default);
"warn": allow but emit a warning. (good for testing)
Now on to res_pjsip...
pjsip_options was modified to retrieve aors with qualify_frequency > 0
rather than all endpoints then all aors. Not only was this a big
improvement in realtime retrieval but even for config files there's an
improvement because we're not going through endpoints anymore.
res_pjsip_mwi was modified to retieve only endpoints with something in
the mailboxes field instead of all endpoints then testing mailboxes.
res_pjsip_registrar_expire was completely refactored. It was retrieving
all contacts then setting up scheduler entries to check for expiration.
Now, it's a single thread (like keepalive) that periodically retrieves
only contacts whose expiration time is < now and deletes them. A new
contact_expiration_check_interval was added to global with a default of
30 seconds.
Ross Beer reports that with this patch, his Asterisk startup time dropped
from around an hour to under 30 seconds.
There are still objects that can't be filtered at the database like
identifies, transports, and registrations. These are not going to be
anywhere near as numerous as endpoints, aors, auths, contacts however.
Back to allow_unqualified_fetch. If this is set to yes and you have a
very large number of objects in the database, the pjsip CLI commands
will attempt to retrive ALL of them if not qualified with a LIKE.
Worse, if you type "pjsip show endpoint <tab>" guess what's going to
happen? :) Having a cache helps but all the objects will have to be
retrieved at least once to fill the cache. Setting
allow_unqualified_fetch=no prevents the mass retrieve and should be used
on endpoints, auths, aors, and contacts. It should NOT be used for
identifies, registrations and transports since these MUST be
retrieved in bulk.
Example sorcery.conf:
[res_pjsip]
endpoint=config,pjsip.conf,criteria=type=endpoint
endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error
ASTERISK-25826 #close
Reported-by: Ross Beer
Tested-by: Ross Beer
Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67
2016-03-08 21:55:30 +00:00
|
|
|
|
2017-09-11 10:46:35 +00:00
|
|
|
return start;
|
|
|
|
}
|